Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions .github/workflows/data/clickhouse/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,21 @@
clickhouse-version: 25.9-alpine
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
clickhouse-image: clickhouse/clickhouse-server
clickhouse-version: 25.9-alpine
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
16 changes: 12 additions & 4 deletions .github/workflows/data/core/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,26 @@
3_5_x: &3_5_x
spark-version: "3.5"
pydantic-version: 2
python-version: '3.13'
python-version: '3.14'
java-version: 20
os: ubuntu-latest

4_0_x: &4_0_x
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
2 changes: 1 addition & 1 deletion .github/workflows/data/ftp/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ min: &min

max: &max
pydantic-version: 2
python-version: '3.13'
python-version: '3.14'
os: ubuntu-latest

matrix:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/data/ftps/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ min: &min

max: &max
pydantic-version: 2
# pydantic v1 fails with `TypeError: cannot pickle 'cell' object` on FTPS, but not on FTP`
python-version: '3.13'
os: ubuntu-latest

Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/data/hdfs/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,20 @@
hadoop-version: hadoop3-hdfs
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
hadoop-version: hadoop3-hdfs
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
14 changes: 11 additions & 3 deletions .github/workflows/data/hive/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,19 @@
4_0_x: &4_0_x
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
13 changes: 11 additions & 2 deletions .github/workflows/data/kafka/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,19 @@
pydantic-version: 2
spark-version: "4.0"
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
kafka-version: 3.9.0
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
14 changes: 11 additions & 3 deletions .github/workflows/data/local-fs/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,19 @@
4_0_x: &4_0_x
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
15 changes: 12 additions & 3 deletions .github/workflows/data/mongodb/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,20 @@
mongodb-version: 8.2.2
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
mongodb-version: 8.2.2
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
15 changes: 12 additions & 3 deletions .github/workflows/data/mssql/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,20 @@
mssql-version: 2025-latest
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
mssql-version: 2025-latest
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
15 changes: 12 additions & 3 deletions .github/workflows/data/mysql/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,20 @@
mysql-version: 9.5.0
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
mysql-version: 9.5.0
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
17 changes: 14 additions & 3 deletions .github/workflows/data/oracle/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,22 @@
db-name: FREEPDB1
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
oracle-image: gvenzl/oracle-free
oracle-version: 23.26.0-slim-faststart
db-name: FREEPDB1
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
15 changes: 12 additions & 3 deletions .github/workflows/data/postgres/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,20 @@
postgres-version: 18.1-alpine
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
postgres-version: 18.1-alpine
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
15 changes: 12 additions & 3 deletions .github/workflows/data/s3/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,20 @@
minio-version: 2025.7.23
spark-version: "4.0"
pydantic-version: 2
python-version: '3.13'
# https://stackoverflow.com/a/79017758/23601543
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

4_1_x: &4_1_x
minio-version: 2025.7.23
spark-version: "4.1"
pydantic-version: 2
python-version: '3.14'
# https://issues.apache.org/jira/browse/SPARK-54276
java-version: 22
os: ubuntu-latest

matrix:
small: [*3_5_x]
full: [*3_2_x, *3_5_x, *4_0_x]
full: [*3_2_x, *3_5_x, *4_0_x, *4_1_x]
2 changes: 1 addition & 1 deletion .github/workflows/data/sftp/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ min: &min
max: &max
openssh-version: 9.9_p2-r0-ls193
pydantic-version: 2
python-version: '3.13'
python-version: '3.14'
os: ubuntu-latest

matrix:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/data/webdav/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ min: &min
max: &max
webdav-version: latest
pydantic-version: 2
python-version: '3.13'
python-version: '3.14'
os: ubuntu-latest

matrix:
Expand Down
18 changes: 10 additions & 8 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Non-goals
Requirements
------------

* **Python 3.7 - 3.13**
* PySpark 3.2.x - 4.0.x (depends on used connector)
* **Python 3.7 - 3.14**
* PySpark 3.2.x - 4.1.x (depends on used connector)
* Java 8+ (required by Spark, see below)
* Kerberos libs & GCC (required by ``Hive``, ``HDFS`` and ``SparkHDFS`` connectors)

Expand Down Expand Up @@ -180,17 +180,19 @@ Compatibility matrix
^^^^^^^^^^^^^^^^^^^^

+--------------------------------------------------------------+-------------+-------------+-------+
| Spark | Python | Java | Scala |
| Spark | Python | Java | Scala |
+==============================================================+=============+=============+=======+
| `3.2.x <https://spark.apache.org/docs/3.2.4/#downloading>`_ | 3.7 - 3.10 | 8u201 - 11 | 2.12 |
| `3.2.x <https://spark.apache.org/docs/3.2.4/#downloading>`_ | 3.7 - 3.10 | 8u201 - 11 | 2.12 |
+--------------------------------------------------------------+-------------+-------------+-------+
| `3.3.x <https://spark.apache.org/docs/3.3.4/#downloading>`_ | 3.7 - 3.12 | 8u201 - 17 | 2.12 |
| `3.3.x <https://spark.apache.org/docs/3.3.4/#downloading>`_ | 3.7 - 3.12 | 8u201 - 17 | 2.12 |
+--------------------------------------------------------------+-------------+-------------+-------+
| `3.4.x <https://spark.apache.org/docs/3.4.4/#downloading>`_ | 3.7 - 3.12 | 8u362 - 20 | 2.12 |
| `3.4.x <https://spark.apache.org/docs/3.4.4/#downloading>`_ | 3.7 - 3.12 | 8u362 - 20 | 2.12 |
+--------------------------------------------------------------+-------------+-------------+-------+
| `3.5.x <https://spark.apache.org/docs/3.5.1/#downloading>`_ | 3.8 - 3.13 | 8u371 - 20 | 2.12 |
| `3.5.x <https://spark.apache.org/docs/3.5.1/#downloading>`_ | 3.8 - 3.13 | 8u371 - 20 | 2.12 |
+--------------------------------------------------------------+-------------+-------------+-------+
| `4.0.x <https://spark.apache.org/docs/4.0.1/#downloading>`_ | 3.9 - 3.13 | 17 - 22 | 2.13 |
| `4.0.x <https://spark.apache.org/docs/4.0.1/#downloading>`_ | 3.9 - 3.14 | 17 - 22 | 2.13 |
+--------------------------------------------------------------+-------------+-------------+-------+
| `4.1.x <https://spark.apache.org/docs/4.1.1/#downloading>`_ | 3.10 - 3.14 | 17 - 22 | 2.13 |
+--------------------------------------------------------------+-------------+-------------+-------+

.. _pyspark-install:
Expand Down
2 changes: 2 additions & 0 deletions docs/changelog/next_release/+.feature.1.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Test Spark 4.1.x compatibility.
For now, Iceberg & Excel packages have no versions support for that, but other packages already supporting Spark 4.0.x can be used freely.
1 change: 1 addition & 0 deletions docs/changelog/next_release/+.feature.2.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Test Python 3.14 compatibility.
2 changes: 1 addition & 1 deletion docs/connection/db_connection/clickhouse/prerequisites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Version Compatibility
* Clickhouse server versions:
* Officially declared: 22.8 or higher
* Actually tested: 21.1, 25.8
* Spark versions: 3.2.x - 4.0.x
* Spark versions: 3.2.x - 4.1.x
* Java versions: 8 - 22

See `official documentation <https://clickhouse.com/docs/en/integrations/java#jdbc-driver>`_.
Expand Down
2 changes: 1 addition & 1 deletion docs/connection/db_connection/hive/prerequisites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Version Compatibility
* Hive Metastore version:
* Officially declared: 0.12 - 3.1.3 (may require to add proper .jar file explicitly)
* Actually tested: 1.2.100, 2.3.10, 3.1.3
* Spark versions: 3.2.x - 4.0.x
* Spark versions: 3.2.x - 4.1.x
* Java versions: 8 - 22

See `official documentation <https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html>`_.
Expand Down
2 changes: 1 addition & 1 deletion docs/connection/db_connection/kafka/prerequisites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Version Compatibility
* Kafka server versions:
* Officially declared: 0.10 or higher
* Actually tested: 3.2.3, 3.9.0 (only Kafka 3.x supports message headers)
* Spark versions: 2.4.x - 4.0.x
* Spark versions: 2.4.x - 4.1.x
* Java versions: 8 - 22

See `official documentation <https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html>`_.
Expand Down
Loading