spark-3.5-spline-agent not working with Glue 5.0 spark 3.5

[pratikmore25]

Hi,

I tried using Spark-3.5-spline-agent-bundle_2.12-2.2.1.jar with AWS Glue 5.0, but encountered an error: IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder'.

However, when I use the following options:

Key: --java-options
Value: --add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED

the job runs successfully, though the runtime increases by approximately 1.5 minutes.

Is there an alternative solution to resolve this issue?

[pratikmore25]

It seems that the different versions of Scala and Java are causing errors when I try to run the Spline jar for Spark 3.5 on AWS Glue 5. Specifically, AWS Glue uses Scala 2.12.18 and JDK 21+, while Apache Spline uses Scala 2.12.0 and Java 8.

[jakipatryk]

We are having similar issue with EMR 7.8.0/Spark 3.5.4/Java 17/Spline Agent 2.2.3. Relevant stack trace:

java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:200)
java.base/java.lang.reflect.Method.setAccessible(Method.java:194)
scala.reflect.package$.ensureAccessible(package.scala:65)
scala.runtime.ScalaRunTime$.ensureAccessible(ScalaRunTime.scala:162)
za.co.absa.spline.commons.lang.ARM$.reflMethod$Method1(ARM.scala:32)
za.co.absa.spline.commons.lang.ARM$.using(ARM.scala:32)
za.co.absa.spline.harvester.conf.YAMLConfiguration.(YAMLConfiguration.scala:37)
za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack$.defaultConfig(StandardSplineConfigurationStack.scala:44)
za.co.absa.spline.harvester.SparkLineageInitializer.createListener(SparkLineageInitializer.scala:123)
za.co.absa.spline.harvester.SparkLineageInitializer$SplineSparkSessionWrapper.enableLineageTracking(SparkLineageInitializer.scala:106)

Edit: seems to be the same issue as #782

[wajda]

This should help:

--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.fs=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED

But I'll leave this issue open for awhile just in case I find a way to avoid using reflection in certain places and cut the issue at its roots.