Incompatible versions between Zeebe broker, Operate and hazelcast when I use DMN in process

[portugalw]

I have a problem when I use DMN in zeebe new versions.

Incompatible versions between Zeebe broker, Operate and hazelcast when I use DMN in process:

Zeebe 8.1.9 Operate 8.1.9 zeebe-hazelcast-exporter 1.2.1 = Works Fine!
Zeebe 8.2.12 Operate 8.2.12 zeebe-hazelcast-exporter 1.2.1 = Doesn't Work!
Zeebe 8.2.12 Operate 8.2.12 zeebe-hazelcast-exporter 1.4.0 = Doesn't Work!
Zeebe 8.3.7 Operate 8.3.7 zeebe-hazelcast-exporter 1.2.1 = Doesn't Work!
Zeebe 8.3.7 Operate 8.3.7 zeebe-hazelcast-exporter 1.4.0 = Doesn't Work!

Every time when the process pass through the DMN, the following error was thrown:

2024-03-14 17:09:02 2024-03-14 20:09:02.428 [Broker-0] [zb-fs-workers-1] [Exporter-1] ERROR
2024-03-14 17:09:02 io.camunda.zeebe.broker.exporter - Actor 'Exporter-1' failed in phase STARTED with: java.lang.NoSuchMethodError: 'long io.camunda.zeebe.protocol.record.value.EvaluatedDecisionValue.getDecisionVersion()' .
2024-03-14 17:09:02 java.lang.NoSuchMethodError: 'long io.camunda.zeebe.protocol.record.value.EvaluatedDecisionValue.getDecisionVersion()'
2024-03-14 17:09:02 at io.zeebe.exporter.proto.RecordTransformer.toEvaluatedDecision(RecordTransformer.java:612) ~[?:?]
2024-03-14 17:09:02 at io.zeebe.exporter.proto.RecordTransformer.toDecisionEvaluationRecord(RecordTransformer.java:571) ~[?:?]
2024-03-14 17:09:02 at io.zeebe.exporter.proto.RecordTransformer.toProtobufMessage(RecordTransformer.java:163) ~[?:?]
2024-03-14 17:09:02 at io.zeebe.exporter.proto.RecordTransformer.toGenericRecord(RecordTransformer.java:174) ~[?:?]
2024-03-14 17:09:02 at io.zeebe.hazelcast.exporter.HazelcastExporter.recordToProtobuf(HazelcastExporter.java:160) ~[?:?]
2024-03-14 17:09:02 at io.zeebe.hazelcast.exporter.HazelcastExporter.export(HazelcastExporter.java:147) ~[?:?]
2024-03-14 17:09:02 at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$export$5(ExporterContainer.java:181) ~[zeebe-broker-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:34) ~[zeebe-util-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.export(ExporterContainer.java:180) ~[zeebe-broker-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:167) ~[zeebe-broker-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:528) ~[zeebe-broker-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:92) ~[zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) [zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) [zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:109) [zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) [zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:205) [zeebe-scheduler-8.2.12.jar:8.2.12]
2024-03-14 17:09:02 2024-03-14 20:09:02.440 [Broker-0] [zb-actors-0] [ZeebePartition-1] WARN
2024-03-14 17:09:02 io.camunda.zeebe.broker.system - Exporter-1 failed, marking it as unhealthy: Exporter-1{status=UNHEALTHY, issue=HealthIssue[message=null, throwable=java.lang.NoSuchMethodError: 'long io.camunda.zeebe.protocol.record.value.EvaluatedDecisionValue.getDecisionVersion()', cause=null]}
2024-03-14 17:13:58 2024-03-14 20:13:58.708 [Broker-0] [zb-fs-workers-0] [SnapshotStore-1] INFO
2024-03-14 17:13:58 io.camunda.zeebe.snapshots.impl.FileBasedSnapshotStore - Committed new snapshot 3584-1-5344-4063

[saig0]

@portugalw thank you for reporting.

The Zeebe Hazelcast Exporter version 1.4.0 should work for Zeebe 8.2.x and 8.3.x.

I tested the compatibility with 8.2.18 and 8.3.5. ✔️

Please verify again with the exporter version 1.4.0.

[portugalw]

This is my compose. There something wrong ?

zeebe: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
image: myregistry/camunda/zeebe:${CAMUNDA_PLATFORM_VERSION:-8.3.5}
container_name: zeebe
ports:
- "26500:26500"
- "9600:9600"
- "5701:5701"
environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1
# allow running with low disk space
- ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
- ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
- "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
restart: always
volumes:
- zeebe:/usr/local/zeebe/data
- ./.compose-configs/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar
- ./.compose-configs/application.yaml:/usr/local/zeebe/config/application.yaml
networks:
- camunda-platform
depends_on:
- elasticsearch

[portugalw]

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-03-18 14:33:50 2024-03-18 17:33:50.499 [] [main] [] ERROR
2024-03-18 14:33:50 org.springframework.boot.SpringApplication - Application run failed
2024-03-18 14:33:50 java.lang.IllegalStateException: Failed to execute CommandLineRunner
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:774) ~[spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:755) [spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:750) [spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.run(SpringApplication.java:323) [spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 Caused by: java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar', className='io.zeebe.hazelcast.exporter.HazelcastExporter', args=null}
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:149) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:70) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:49) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 ... 12 more
2024-03-18 14:33:50 Caused by: io.camunda.zeebe.broker.exporter.repo.ExporterLoadException: Cannot load exporter [hazelcast]: cannot load specified class
2024-03-18 14:33:50 at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:82) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:147) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:70) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:49) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 ... 12 more
2024-03-18 14:33:50 Caused by: java.lang.ClassNotFoundException: io.zeebe.hazelcast.exporter.HazelcastExporter
2024-03-18 14:33:50 at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
2024-03-18 14:33:50 at io.camunda.zeebe.util.jar.ExternalJarClassLoader.loadClass(ExternalJarClassLoader.java:57) ~[zeebe-util-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:79) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:147) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:70) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.Broker.(Broker.java:49) ~[zeebe-broker-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.5.jar:8.3.5]
2024-03-18 14:33:50 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.6.jar:3.1.6]
2024-03-18 14:33:50 ... 12 more
2024-03-18 14:33:50 2024-03-18 17:33:50.505 [] [main] [] WARN
2024-03-18 14:33:50 org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent

[saig0]

@portugalw, please ensure that the JAR is located correctly and matches the configuration.

The following exception message indicated that the JAR doesn't exist.

2024-03-18 14:33:50 Caused by: io.camunda.zeebe.broker.exporter.repo.ExporterLoadException: Cannot load exporter [hazelcast]: cannot load specified class
...
2024-03-18 14:33:50 Caused by: java.lang.ClassNotFoundException: io.zeebe.hazelcast.exporter.HazelcastExporter