Drop Java 8 support and move to Java 11/17?

[vinothchandar]

Could we skip 11 and go directly to 17? Flink only supports Java 17 from 1.18 onwards?

Thoughts? Considerations? Is anyone still on Java 8 - or have objections to dropping java 8 ..

Here's some WIP PR on moving to 11. any takers to get this to finish line (assuming we are okay doing so as a group)
#12880

[vinothchandar]

Spark versions that can work with Java 17.

• Spark 4: Java 17+
• Spark 3.3 : supports 8/11/17

Presto/Trino have moved onto Java 21+ or so..

So its just Flink. that will drag us back to Java 11.

What flink version is most prevalently used with Hudi? @danny0405 ?

[vinothchandar]

@voonhous says Flink CDC does not work with Java 17 yet?

[voonhous]

From what i can see, Java 17 is not tested yet. The available profiles on Flink-CDC are 8 and 11.

[vinothchandar]

From @yihua on the dev sync.

Spark - Java 17
Flink - Java 11, Java 17 from 1.18
Trino - Java 22
Hive - Java 8, 11/17 may work but only with latest Hadoop (we're upgrading hadoop anyway)

[cshuo]

Flink supports java17 from 1.15, https://issues.apache.org/jira/browse/FLINK-25005

[rangareddy]

Hive and Java Compatibility:

Hive Version	Java Version	Hadoop Version
Hive 3.1.0/3.1.1	Java 7 or Java8	Hadoop 3.1.0
Hive 3.1.2/3.1.3	Java 8	Hadoop 3.1.0
Hive 4.0.1	Java 8	Hadoop 3.3.6
Hive 4.1.x	Java 17	Hadoop 3.4.1
Hive 4.2.x	Java 21	Hadoop 3.4.1

[voonhous]

Had a sync with @cshuo and @danny0405 today and we have came up with the following conclusion:

If Java 17 and Java 11 is compatible, we can upgrade to Java 17 directly.

However, most Chinese users JDK 8, we might need to provide users with a guide on how to perform manual compilation JDK8.

[vinothchandar]

So - we cannot drop java 8 support? There is little benefit from moving to Java 11/17, if we cannot use the language features, right

[danny0405]

Our repo does not utililize the Java 11/17 features though, it's great if we can stick to Java8 still, I got the impression we must move to either Java 11 or 17 so I prefer 17 there without realizing 8 is still an option.

[geserdugarov]

So - we cannot drop java 8 support? There is little benefit from moving to Java 11/17, if we cannot use the language features, right

There are some performance improvements. I don't have any numbers but know that Hudi 0.15 moved from Java 8 to Java 17 perform better without using any new features. Note, that it was Flink stream writing use case, and current master in comparison with 0.15 differs a lot.

[cshuo]

Java 8 bytecode can also run in JRE 11/17, so it can also benefit from the performance gain of newer JDK, such as optimizations for JIT, GC..

[vinothchandar]

Hudi 0.15 moved from Java 8 to Java 17 perform better without using any new features

Same as @cshuo 's point here.. We can still run java 8 byte code on higher version JREs for this benefit right?

[xushiyan]

+1 to move to java 17, and providing a java 8 compilation guide.

[vinothchandar]

@xushiyan @voonhous Please clarify my question above. What's the benefit of the move?

[vinothchandar]

My question still is : what do we gain by making java 17 as default if we stick to java 8 language level. How'd we take advantage of the latest language features.

I'd love to drop support for Java 8 and move to Java 11 or 17.

[vinothchandar]

@danny0405 @voonhous Are we saying we are keeping code language level at 8?

[danny0405]

yeah, because 8 still got a lot of users in production, not introducing new syntax beyond would help for the compatibility.

[vinothchandar]

To summarize: we should either drop support for Java 8 or stay at Java 8 as default.

Looks like we are saying there are still Flink users on java 8, that we'd like to support

[vinothchandar]

@voonhous @cshuo @danny0405 - can we circulate this discussion on Flink forums and see if users are ok with dropping java 8? I'd like to def drop java 8 and move language level to Java 11 atleast in 2.0

[voonhous]

Will followup on this.

[vinothchandar]

Unless we hear strong appeals, I propose to end java 8 support with the 1.x release line. There are lots of latest features in jdk, for the RFC-100 type work.

[danny0405]

Unless we hear strong appeals, I propose to end java 8 support with the 1.x release line. There are lots of latest features in jdk, for the RFC-100 type work.

we prefer to skip JDK 11 and support JDK 17 directly right? +1 for that.

[suryaprasanna]

Unless we hear strong appeals, I propose to end java 8 support with the 1.x release line. There are lots of latest features in jdk, for the RFC-100 type work.

we prefer to skip JDK 11 and support JDK 17 directly right? +1 for that.

+1 for JDK 17 as well. Some of the MCP server dependency are compiled with Java 17. So, it will be good to move to JDK 17.

[yihua]

+1 on ending JDK 8 support from the next major 1.x release. Given recent findings that RFC-100 (unstructured data support in Hudi with file format including Lance) requires JDK 11 (#14128), keeping Java 8 support blocks new features and future development. Also, users requiring Java 8 can still upgrade to Hudi 1.1.

I propose that we move to JDK 11 as the minimum for Hudi 1.2 release as a first step, and then move to JDK 17 (which requires Scala version upgrade and other major changes based on my previous experience) as the minimum in another release.

[rahil-c]

+1

[voonhous]

+1

[suryaprasanna]

+1

[CTTY]

+1, on AWS we are already on Java 17 by default

[mansipp]

+1

[vinothchandar]

We have voted on the DEV list to drop java 8, in 1.2.

[xushiyan]

To clarify the support matrix, in the next release Hudi 1.2, since we're dropping java 8, moving to 11 or 17 has different implications of engine integration compatibility. Below is a support matrix:

Engine	Versions	Min Runtime JVM	Java 11 Compile Target	Java 17 Compile Target	Notes
Spark	3.3.x, 3.4.x, 3.5.x	Java 8 (11 recommended)	✅ Compatible	⚠️ Requires Java 17 runtime
Spark	4.x	Java 17	✅ Compatible	✅ Compatible	Scala 2.13 only.
Flink	1.18.x, 1.19.x, 1.20.x	Java 8 (11 recommended)	✅ Compatible	⚠️ Requires Java 17 runtime (beta)	Java 17 is beta in 1.18–1.20.
Flink	2.0.x	Java 11 (17 recommended)	✅ Compatible	⚠️ Requires Java 17 runtime	Java 17 default; Java 11 minimum.
Hive	3.x (EOL)	Java 8	⚠️ Incompatible	⚠️ Incompatible	no official support for running with java11+ runtime; may work but only with latest Hadoop
Hive	4.0.x	Java 8	⚠️ Incompatible	⚠️ Incompatible	no official support for running with java11+ runtime; may work but only with latest Hadoop
Hive	4.1.x	Java 17	✅ Compatible	✅ Compatible
Hive	4.2.x	Java 21	✅ Compatible	✅ Compatible
Trino	472+	Java 23/24	✅ Compatible	✅ Compatible
PrestoDB	0.273–0.286	Java 11	✅ Compatible	⚠️ Requires Java 17 runtime
PrestoDB	0.287+	Java 17	✅ Compatible	✅ Compatible

Based on flink 1.18-1.20 only having beta support to work with java 17, moving to java 11 is safer. Also note that Hive support needs further verification, going with java 11 or 17 would require extra setup or dropping support before hive 4.1.

I propose moving to java 11 as default language level and compile target in Hudi 1.2, this means dropping flink 1.17, hive 3.x & 4.0.x support (if no hadoop upgrade workaround).