v0.1.0-beta.5

Docker Images

Both images support linux/amd64 and linux/arm64.

Minimal (scratch-based):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.5
docker pull vnvohub/deltaforge:0.1.0-beta.5

Debug (includes shell):

docker pull ghcr.io/vnvo/deltaforge:0.1.0-beta.5-debug
docker pull vnvohub/deltaforge:0.1.0-beta.5-debug

What's Changed

• updates the readme and the docs with the latest on turso and schema-sensing (57741f1)
• Merge pull request #32 from vnvo/feat/tursosource (0c0c889)
• ignore mysql e2e test by default - to prevent ci failure (be03258)
• fix clippy complaints (62fce34)
• fotmatting fixes (db3ea2e)
• calculate the column count per schema version (00270ff)
• update dev.sh and example configs (f3002f6)
• adds sensing cache stats (7a874de)
• make the pipeline_manager simpler and separate all the controller implementations (89e2bb2)
• adds get and delete endpoint for pipelines (cc4d724)
• optimize the schema sensing integration in the critical path (2c8eb22)
• updates bench test for pipeline end to end execution (4ba5140)
• check the required field for sinks (98be96d)
• implement the required check and config cover for the sinks (9a1cd95)
• wire schema provider into coordinator for guided schema sensing (52be71c)
• turso source with some tests (f51b3ac)
• initial integration of schema sensing into runner (0a41b33)
• dedicated schema-sensing config and better naming (9383f67)
• integrates schema inference with the runner (544f9af)
• adds schema-sensing module to infer json schemas in flight (8ccea69)
• adds turso srouce config parsing (1b39693)
• update the readme and docs with the latest schema and checkpointing semantics (8977b17)
• Merge pull request #30 from vnvo/fix/checkpointing (d6ffd00)
• remove unused arc (184eeca)
• revert to using Option to avoid custom serde shenanigans (8ea7673)
• allow debug symbols for benching (f267610)
• set both schema_version and sequence using arc (53ad335)
• fix js processor test for added senquence (4e511f4)
• set the schema senquence from the source (333aaa3)
• update the schema finger-printing tests (c54841e)
• make schema fingerprints stable using sha265 (40f750e)
• fix checkpoints inconsistency between source and runner (d9c3442)
• Merge pull request #28 from vnvo/df-data (c82a272)
• formating fixes (1647564)
• fixing rusqlite incompatiblity with usize (6b045e4)
• cleanup the the test and adds relevant docs for schema registry (cdee4c7)
• expose schema functionality via rest apis (b9fed82)
• integrate new schema handling into mysql mod and update the e2e tests (41668d1)
• feat: extends the existing schema registry and checkpoints with versioning and more backend options (ba95c95)
• Merge pull request #26 from vnvo/hotpath (0d9b027)
• formatting fixes (5726358)
• perf(core): use Arc<[u8]> for checkpointmeta and pipeline bench tests (ff2d84a)
• feat: adds proper batch send api for sinks and removes unnecessary clones (8d923d9)
• update the logging and o11y docs (297ba38)
• Merge pull request #25 from vnvo/metrics (24d6295)
• ignore clippy complaints on too many args (ec7ed5f)
• optimize event size calc and pipeline name passing in metrics and fixes the tests (a8de2f2)
• extends the metrics and provides a guide about them (6f583ea)
• updates readme and docs with new dev details (be6c370)

What's Changed

• Metrics by @vnvo in #25
• Hotpath by @vnvo in #26
• Df data by @vnvo in #28
• Fix/checkpointing: schema-checkpoint correlation + Event metadata population by @vnvo in #30
• Feat/tursosource by @vnvo in #32

Full Changelog: v0.1.0-beta.4...v0.1.0-beta.5