Initialization failure handling control #757
Description
Add a configuration property to control how the agent should behave on initialization failures.
Currently when an error occurs during initialization phase of the agent (e.g. misconfiguration, failed handshake with the server etc) the error is logged and the Spark job carries on.
23/10/23 17:09:34 ERROR SparkLineageInitializer: Spline initialization failed! Spark Lineage tracking is DISABLED.
...
23/10/23 17:09:35 INFO CodeGenerator: Code generated in 77.142417 ms
23/10/23 17:09:35 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 198.9 KiB, free 4.6 GiB)
23/10/23 17:09:36 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 33.9 KiB, free 4.6 GiB)
23/10/23 17:09:36 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:59667 (size: 33.9 KiB, free: 4.6 GiB)
23/10/23 17:09:36 INFO SparkContext: Created broadcast 0 from csv at CodelessInitExampleJob.scala:34
23/10/23 17:09:36 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
23/10/23 17:09:36 INFO SparkContext: Starting job: csv at CodelessInitExampleJob.scala:34
23/10/23 17:09:36 INFO DAGScheduler: Got job 0 (csv at CodelessInitExampleJob.scala:34) with 1 output partitions
23/10/23 17:09:36 INFO DAGScheduler: Final stage: ResultStage 0 (csv at CodelessInitExampleJob.scala:34)
23/10/23 17:09:36 INFO DAGScheduler: Parents of final stage: List()
Such behavior was chosen with the aim to not affect the Spark job and do not interrupt potentially higher priority (from the operational perspective) processes. But sometimes, when lineage is mandatory for the user, they might prefer the Spark job to explicitly fail instead of silently continue without lineage tracking.
This behavior can be controlled by the config property:
spline.onInitFailure = LOG | BREAKThe default value would be LOG that corresponds to the current behavior. The BREAK mode would simply propagate the error to the Spark process causing the Spark job to fail.