v0.1.0-beta.7

Docker Images

Both images support linux/amd64 and linux/arm64.

Minimal (scratch-based):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.7
docker pull vnvohub/deltaforge:0.1.0-beta.7

Debug (includes shell):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.7-debug
docker pull vnvohub/deltaforge:0.1.0-beta.7-debug

What's Changed

• release: v0.1.0-beta.7 (79ef40a)
• Merge pull request #45 from vnvo/dyn-routing (e892e67)
• changelog updated (5ec5342)
• adds landing page for deltaforge (61f7c1b)
• updated docs for routing and smaller corrections (cb6bafb)
• add smaple config for dynamic routing (81cf409)
• fix formatting (12e6d2f)
• make env var expansion not to fail and treat it as a template instead (e2b185b)
• refactor sink integration tests and add routing test cases (75dae63)
• more tests for js routing overrides (aed8b59)
• adds dynamic routing support to js processor (0c6efda)
• clean up dead code (6fe2709)
• update broken unit tests related to sinks (c8e23cd)
• update sink integration tests (43dfb16)
• update sinks to resolve topic and keys and handle the headers when supported (f164f5b)
• update kafka sink to resolve the topic and key dynamically and handle the headers (61e782c)
• add EventRouting struct and CompiledTempalte for dynamic sink dest (52a1d54)
• rpelace sscache with swatinem/rust-cache@v2, use cargo-binstall for all tool instances and some cleanups (ba5dcde)
• improve gha workflows for proper caching and to prevent target bloat (95bdba9)
• skip the buggy flowstat version (4dde50e)
• Merge pull request #42 from vnvo/schema-sensing-hc (656e9e8)
• fix clippy stuff (f2cf0f7)
• fix formatting issues (af17011)
• update the changelog (782bae4)
• docs(schema-sensing): update all the docs related to schema-sensing (5a8d688)
• refactor(schema-sensing): cleanup the bench tests, add metrics (2c32ed5)
• bench and unit tests for high cardinality schema sensing and related optimizations (655a31c)
• schema-sensing integration with high cardinality handling (5addfa5)
• refactor(schema-sensing): adds high_cardinality mod using flowstats crate (5aa1482)
• Merge pull request #39 from vnvo/envelop (af0710d)
• changelog updated (07673c2)
• docs: updated the readme and the example configs (d39f455)
• refactor: debezium envelope has schema field but with null value for now (0e5cf38)
• refactor: actually use the encoding functionality in sinks (c4d5ac7)
• docs: added envelope and encoding and more examples (538eee6)
• fixing clippy issues and removed turso integration tests for now (17d933c)
• refactor(tests): updated all tests to integrate envelope and encoding (6938491)
• fix(sources-mysql): fix ddl handling and required schema reload (6dfe592)
• refactor(runner): integrate envelope and encoding (867e005)
• refactor(sinks): integrate envelope and encoding into sinks (fce2562)
• refactor(source-postgres): adopt the envelop/encode model (a9ac527)
• refactor(source-mysql): adopt the envelop/encode model (684b165)
• refactor: adds ecoding and envelope configs to sinks (72a1898)
• feat(encoding): added new encoding module (316d763)
• feat(envelope): added new envelope module (71f4fdb)
• refactor(core): adopt debezium envelop as the base event structure (57f2b9c)

What's Changed

• Envelope and Encoding support by @vnvo in #39
• Schema sensing hc by @vnvo in #42
• Dyn routing by @vnvo in #45

Full Changelog: v0.1.0-beta.6...v0.1.0-beta.7

v0.1.0-beta.6

Docker Images

Both images support linux/amd64 and linux/arm64.

Minimal (scratch-based):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.6
docker pull vnvohub/deltaforge:0.1.0-beta.6

Debug (includes shell):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.6-debug
docker pull vnvohub/deltaforge:0.1.0-beta.6-debug

What's Changed

• release changelog (b694c08)
• updated changelog (d29f6a0)
• docs: updated sink docs: - NATS sink doc added. - Kafka and Redis sink docs updated with failure modes and other details - Sink's readme updated with details about multi-sink setups and commit policy. (39e32c7)
• feat(nats): NATS source with tests (21d7800)
• fixing format issues, again! (6e77b0e)
• updated changelog (5bf2ccc)
• clippy complaints (d607ef4)
• formatting fixes (1f1b136)
• refactor(sinks): use common utilities in sinks and revamp the tests (85b411e)
• update pgwire-replication dep (c2cea34)
• chore: fixing clippy warnings (db2123a)
• refactor(sources): remove conn_utils (ad9c2ef)
• refactor(source-postgres): use the common utilities and split the tests (526b042)
• fixing clippy complaints (e0ddd56)
• refactor(mysql-source): use new common utilities and update the tests (066d0b3)
• refactor(common): add shared utilities for source, sinks and etc (e6b1eb9)
• update the readme (94ff4e1)
• changelog setup (1a9f54a)
• adds version extraction and printing (de27d74)
• add postgres to the summary links too (72e01c8)
• adds libssl and libsasl to the debug image (a1785c2)
• always copy ssl certs from build phase (6e129fc)
• hide turso behind a feature flag, dont CI/Release for all features (696d840)
• fmt fixes (9828428)
• clean/hide unused stuff, fix clippy complaints (73d2af3)
• fix clippy complaint (cc788b6)
• apply workspace linting to all crates and related cleanups (7d23d5f)
• update the docs and include postgres (ce9f6a2)
• fixed list_cached call in the mysql cdc test (3875431)
• properly handle the retry for mysql source (501a418)
• adds binlog_row_image check and removes extra list_cached method (39c86f4)
• retry backoff tests (c613fa0)
• make pg config simpler and cover with tests (9d98142)
• handle retires for postgres (38e879f)
• clean up retrypolicy, re-adjust the default params (8549899)
• bug fix, handle pg array boolean (c6dc3c4)
• complete pg source, dev setup and tests (72ac4bc)
• clippy fixes (4b14209)
• formatting fixes (e80eb6e)
• simplify the turso support and mark it as experimental (729bfe7)
• remove turso non-native modes (c2a399f)
• properly handle process failures (4b23164)
• sample dev configs for turso and simple sqlite (9998b5e)
• fixs checkpointing, coordinator should save it (54c98b2)
• add timeout and is-alive check for js processor (11a89bc)
• don't swallow the js process errors (2ba29d1)
• prevent checkpoint and size_bytes serialization for events (9903c69)

Full Changelog: v0.1.0-beta.5...v0.1.0-beta.6

v0.1.0-beta.5

Docker Images

Both images support linux/amd64 and linux/arm64.

Minimal (scratch-based):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.5
docker pull vnvohub/deltaforge:0.1.0-beta.5

Debug (includes shell):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.5-debug
docker pull vnvohub/deltaforge:0.1.0-beta.5-debug

What's Changed

• updates the readme and the docs with the latest on turso and schema-sensing (57741f1)
• Merge pull request #32 from vnvo/feat/tursosource (0c0c889)
• ignore mysql e2e test by default - to prevent ci failure (be03258)
• fix clippy complaints (62fce34)
• fotmatting fixes (db3ea2e)
• calculate the column count per schema version (00270ff)
• update dev.sh and example configs (f3002f6)
• adds sensing cache stats (7a874de)
• make the pipeline_manager simpler and separate all the controller implementations (89e2bb2)
• adds get and delete endpoint for pipelines (cc4d724)
• optimize the schema sensing integration in the critical path (2c8eb22)
• updates bench test for pipeline end to end execution (4ba5140)
• check the required field for sinks (98be96d)
• implement the required check and config cover for the sinks (9a1cd95)
• wire schema provider into coordinator for guided schema sensing (52be71c)
• turso source with some tests (f51b3ac)
• initial integration of schema sensing into runner (0a41b33)
• dedicated schema-sensing config and better naming (9383f67)
• integrates schema inference with the runner (544f9af)
• adds schema-sensing module to infer json schemas in flight (8ccea69)
• adds turso srouce config parsing (1b39693)
• update the readme and docs with the latest schema and checkpointing semantics (8977b17)
• Merge pull request #30 from vnvo/fix/checkpointing (d6ffd00)
• remove unused arc (184eeca)
• revert to using Option to avoid custom serde shenanigans (8ea7673)
• allow debug symbols for benching (f267610)
• set both schema_version and sequence using arc (53ad335)
• fix js processor test for added senquence (4e511f4)
• set the schema senquence from the source (333aaa3)
• update the schema finger-printing tests (c54841e)
• make schema fingerprints stable using sha265 (40f750e)
• fix checkpoints inconsistency between source and runner (d9c3442)
• Merge pull request #28 from vnvo/df-data (c82a272)
• formating fixes (1647564)
• fixing rusqlite incompatiblity with usize (6b045e4)
• cleanup the the test and adds relevant docs for schema registry (cdee4c7)
• expose schema functionality via rest apis (b9fed82)
• integrate new schema handling into mysql mod and update the e2e tests (41668d1)
• feat: extends the existing schema registry and checkpoints with versioning and more backend options (ba95c95)
• Merge pull request #26 from vnvo/hotpath (0d9b027)
• formatting fixes (5726358)
• perf(core): use Arc<[u8]> for checkpointmeta and pipeline bench tests (ff2d84a)
• feat: adds proper batch send api for sinks and removes unnecessary clones (8d923d9)
• update the logging and o11y docs (297ba38)
• Merge pull request #25 from vnvo/metrics (24d6295)
• ignore clippy complaints on too many args (ec7ed5f)
• optimize event size calc and pipeline name passing in metrics and fixes the tests (a8de2f2)
• extends the metrics and provides a guide about them (6f583ea)
• updates readme and docs with new dev details (be6c370)

What's Changed

• Metrics by @vnvo in #25
• Hotpath by @vnvo in #26
• Df data by @vnvo in #28
• Fix/checkpointing: schema-checkpoint correlation + Event metadata population by @vnvo in #30
• Feat/tursosource by @vnvo in #32

Full Changelog: v0.1.0-beta.4...v0.1.0-beta.5

v0.1.0-beta.4

Docker Images

Both images support linux/amd64 and linux/arm64.

Minimal (scratch-based):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.4
docker pull vnvohub/deltaforge:0.1.0-beta.4

Debug (includes shell):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.4-debug
docker pull vnvohub/deltaforge:0.1.0-beta.4-debug

What's Changed

• multi-arch docker images (amd64/arm64) with debug variant (e7c5f77)

Full Changelog: v0.1.0-beta.2...v0.1.0-beta.4

v0.1.0-beta.1

What's Changed

• Batch support - #4 by @vnvo in #6
• refactor: typed error models across sources, sinks, etc by @vnvo in #8
• introduce new batch-based JsProcessor with persistent runtime by @vnvo in #10
• drop the deltaforge- prefix for most crates and tidy things up by @vnvo in #12
• allow users to override kafka client conf by @vnvo in #15
• Fix tests across multiple crates by @vnvo in #18
• pipeline mgmt apis by @vnvo in #19
• Docs by @vnvo in #20
• adds umami analytics to the user guide by @vnvo in #21
• updates the doc strucutre and theme and with logos and etc by @vnvo in #22
• docker image handling by @vnvo in #24

New Contributors

• @vnvo made their first contribution in #6

Full Changelog: https://github.com/vnvo/deltaforge/commits/v0.1.0-beta.1