Skip to content

feat: add custom data-converter support#85

Open
raszi wants to merge 14 commits intomanetu:masterfrom
metrictech:feat/data-converter
Open

feat: add custom data-converter support#85
raszi wants to merge 14 commits intomanetu:masterfrom
metrictech:feat/data-converter

Conversation

@raszi
Copy link
Contributor

@raszi raszi commented Jan 26, 2026

Overview

This PR configures a custom data converter by implementing the Nippy encoder as a DataConverter rather than hard-wiring it in the codebase.

Previously, the Clojure SDK could only read and write Nippy-encoded payloads, making cross-language interoperability impossible.

By adopting Temporal's standard DataConverter mechanism with support for both Nippy and JSON formats, workflows and activities can now be invoked across different languages, including calling Clojure implementations from other languages.

A follow-up PR will make this interoperability more convenient by adding a mechanism to select an encoder instead of using Nippy per default.

Key Changes

New Custom Data Converter Architecture

  • Added modular converter namespaces under temporal.converter.*:
  • temporal.converter.payload - Core payload construction utilities
  • temporal.converter.byte-string - ByteString conversion helpers
  • temporal.converter.metadata - Payload metadata handling
  • temporal.converter.optional - Optional conversation helpers
  • temporal.converter.nippy - Nippy-based binary serialization (binary/plain)
  • temporal.converter.json - JSON serialization using jsonista (json/plain)
  • temporal.converter.default - Default converter combining Null, Nippy, and JSON converters

Client Integration

  • Updated temporal.client.options to automatically use the custom data converter when none is explicitly provided
  • The :data-converter option remains available for users who want to provide their own implementation

Code Improvements

  • Switched converter implementations from reify to defrecord for better performance and debugging
  • Removed unnecessary atom usage in dev-resources utilities
  • Fixed obsolete function calls and typos
  • Added _ prefix to unused bindings
  • Updated lein-cljfmt dependency

Test Coverage

  • Added comprehensive tests for all new converter namespaces
  • Added tests for client options

Backward compatibility

This change is fully backward compatible. The previous implementation used Nippy for serialization, handling it directly in temporal.internal.utils (e.g., ->objarray, ->args, complete-invoke) and temporal.client.core.

The new implementation consolidates this serialization logic into a dedicated DataConverter architecture (temporal.converter.nippy with encoding type binary/plain), while maintaining the same wire format.
Existing workflows and activities with in-flight data will continue to work without any migration.

This refactoring aligns with Temporal's official SDKs (Java, Python), which all use the data converter mechanism for serialization.

Fixes #28
Fixes #43

@raszi raszi force-pushed the feat/data-converter branch from 637b320 to a702482 Compare January 26, 2026 22:49
raszi added 12 commits January 27, 2026 00:00
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
@raszi raszi force-pushed the feat/data-converter branch from a702482 to 04cb1ef Compare January 26, 2026 23:00
(defn assoc-default-data-converter [{:keys [data-converter] :as params}]
(cond-> params
(not data-converter)
(assoc :data-converter (default-data-converter/create))))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not a singleton instance via a plain def? Why recreate it every call?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about it, but I followed the Java SDK logic there. I can definitely create that, I can also create a namespace called temporal.converter and create it there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this shouldn't matter much since this function won't be called often.

Signed-off-by: KARASZI István <ikaraszi@gmail.com>
Signed-off-by: KARASZI István <ikaraszi@gmail.com>
@raszi raszi marked this pull request as ready for review January 28, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross-language workflows Readable encoding (i.e. not nippy)

3 participants