v0.11.6 - Fix Memory safety issue

0.11.6

Fixes

• Fix possible memory safety issue when using patch or merge on state #1217.

Affected tremor-script code sample:

# this is where the memory safety issue occured
let state = merge state of event end;
emit state;

As an optimization we were optimizing both patch and merge to update the target expression (state in the case above) in-place, but only if the merge/patch result is assigned to itself. In case of state as target expression we were adding references to a given memory region where the event data was allocated and those references now outlived the actual event data allocation. 😱

This whole optimization has been removed for now.

Release v0.11.5

0.11.5

Fixes

• Fix artefact sorting in tremor startup script

v0.11.4 - Fixing the release builds

Fixes

• Fix the Release archive and package build process

v0.11.3 - Fixing our Release builds

New features

• Add tests for merge feature in tremor-script #721

Fixes

• Remove a number of transmutes
• Catalog remaining transmutes with related tickets
• Bump snmalloc-rs to 0.2.26 to ensure builds succeed on systems with libc < 2.25

v0.11.2

New features

• Change format of KV offramp responses to a more unified structure.
• Add op key to KV offramp responses in order to differentiate responses by the command that triggered them.
• Add KnownKey::map_* functions to directly work on the Value::Objects inner HashMap, if available.
• Add HEALTHCHECK to Dockerfiles
• Improve printing for dot files.
• Scan sub directories of ${CFG_DIR} for config files.
• Add offramp and onramp for AMQP with the lapin package.

Fixes

• KV offramp sends error responses for each failed command
• Ensure binding to first bind linked offramps to pipelines, then pipelines to offramps/onramps/pipeline, then onramps to pipelines to ensure events only start flowing when all downstreams are connected.
• Fix empty record pattern to only match records.
• Fix ws offramp not reconnecting after connection loss.
• Run tests in tremor-cli bin
• Switch operations of tremor dbg lex and tremor dbg preprocess as they did the job of the other.
• Fix heredoc preprocessing, which was messing up error reporting
• Fix false positives in cycle detection
• Include cncf::otel stdlib sources in deb package
• Add /usr/local/share/tremor to default TREMOR_PATH also for all packages as a well-known directory for custom tremor-script libraries and modules.
• Record the partition number assigned during rebalancing when running Kafka.
• Fix bug in HDR histogram implementation when using emit without reset.
• Fix bug in mean that invalid values would be counted as part of the total number of values.
• Avoid possible contraflow cycles via system::metrics pipeline, if a pipeline is connected to an output port of system::metrics pipeline.

v0.11.1

This release contains several bugfixes and lots of new features, like a GCP Cloud Storage sink, a syslog codec and a dns sink.

We want to hightlight the syslog codec and the GCP sink, as these are contributions from our LFX spring mentorship mentees. Great job!

New features

• Add tremor_value::structurize convenience fn
• Change qos::wal operator to only require one of max_elements or max_bytes (using both is still possible).
• Add DNS sink
• Add syslog codec.
• Add $udp.host and $udp.port to allow controling udp packet destinations on a per event basis.
• Deprecate udp.dst_* config, introduce udp.bind.* config instead.
• Allow insights/contraflow events to traverse through multiple connected pipelines
• Add GCP Cloud Storage linked sink connector.

Fixes

• Fix CI runners to work with caching
• Fix dependencies for tremor-value and tremor-common
• Fix docker entrypoint not forwarding arguments to tremor binary, unless -- is used before them
• Fix match default clause only executing the last statement in the block.
• Kafka back to async with a timeout on waiting
• Fix qos::wal operator never cleaning up the last event from its storage
• Fix generic::batch operator swallowing the transactional status of an Event
• Fix windowed select queries not tracking the transactional status of an Event
• Fix windowed select queries not tracking the Events that constitute an outgoing aggregated event
• Avoid several offramps to swallow fail insights.
• Send correlation metadata for send error events in elastic sink

v0.11.0

New features

• CNCF OpenTelemetry source, sink and cncf::otel tremor-script library adds log, trace, metrics OpenTelemetry support
• Fixes module path function resolution for tremor-cli doc tool to use unified path resolution
• Removed the vsn.sh script which checks if the lockfile is up to date and replaces it with the --locked flag #798
• Allow using '_' as seperators in numeric literals #645
• Refactor kafka metadata variables to be under a single record $kafka.
• Add support for Kafka message headers, available through the $kafka.headers metadata variable.
• Add the cb offramp for testing upstream circuit breaker behaviour #779
• Add the kafka onramp config retry_failed_events to acoid retrying failed events, and polling_interval to control how often kafka is polled for new messages if none were available previously #779
• Add kv connector with the supported operations put, get, delete, scan, cas.
• Add discord badge to README.md.
• Handle signals and terminate properly on Ctrl+C in docker #806
• Update to rust 1.50.0
• Emit error events to the err port on exceeding concurrent requests limit for rest and elastic offramps.
• Add the max_groups and emit_empty_windows settings on window definitions #828
• Restrict event and event metadata references in the SELECT clause of a windowed select statement #828
• Improve default visibility of tremor info logs from packages #850
• Include the request info of a response for a linked rest offramp, available through the $response.request metadata variable.
• Emit warnings when a window with emit_empty_windows without guards is used.
• Extend match to handle top-level ~ extractors in match #834
• Extend match to allow more assign expressions (i.e. case a = ~ glob|snot*| => or case v = _ =>)
• Match pipeline improvements: tree search for ==, grouping based on shared keys, re-ordering of exclusive case statements
• Optimize glob matches of the form glob|snot*| or glob|*badger| to cheaper prefix and suffix checks
• Remove warnings for match w/o default if a _ case or a v = _ case exists
• Add --exprs-only to dbg ast to not show metadata
• Add Delete and Update to ES sink #822
• Add more metadata to Kafka source #874
• Update to simd-json 0.4
• Add tests covering basic operations and string interpolation for tremor-script#721
• Add offramp and onramp for NATS.io.
• Add a stdin onramp.
• Add tests covering arrays and records for tremor-script#721
• Support for non caching UDP sink #900

Fixes

• Fix kafka onramp hanging with no message in the queue, leading to delayed offset commits #779
• Fail the kafka onramp if any of the configured topics could not be subscribed to #779
• Tremor no longer requires a home dir for operations that do not need a config #782
• Add performance section to PR template.
• Fix markdown for discord link.
• Fix lalrpop builds when extra folders exist.
• Refactor operator metrics collection to eliminate clones in most cases.
• Fix systemd spec file to load tremor files in /etc/tremor/config #784
• Do not crash when we can not execute a config file #792
• Sort the artefacts while running the benchmarks so that the benchmark run is more deterministic #825
• Remove the bench_pipe_passthrough_csv benchmark #825
• Fix a bug in the test bench command that was giving false negatives when the benchmarks were failing #816
• Fix time based windows to not emit empty windows anymore and to not error if they contain event references #828
• Do not commit an empty topic-partition-list in the kafka onramp and improve logging for better debugging
• Fix kafka consumer offset lag of at least 1 continually by using offset + 1 when committing.
• Fix issue where binary not (!) was not getting lexed correctly #833
• Fix missing ack/fail insight events with offramps that dont support guaranteed delivery (e.g. udp, stdout) #870
• Fix wrong error message for misconfigured UDP sinks

v0.10.2

0.10.2

Fixes

• Ensure blaster sends all events from the source #759
• Allow the use of const and custom functions using const in select queries #749
• Print hygenic errors when invalid trickle files are loaded in server run -f ... #761
• Ensure elastic offramp does not issue empty bulk requests.
• Avoid sending empty batches from the batch operator.
• Ensure elastic offramp includes event payload in every error response.

New features

• Add discord connector to allow communicating with the discord API

v0.10.1 - Fixing tremor-script crate

Fixes

• Update tremor-value to 0.2 to include binary changes and thus unbreak the 0.10 tremor-script crate

v0.10.0

0.10.0

Breaking Changes

• String interpolation is now done via #{..} instead of {}. { no longer needs to be escaped, but \#{ needs an escape for literal #{.
• Emit tumbling windows based on size immediately when they are full. #731
• Emit tumbling windows based on time interval also when no event comes in but the interval passed #731
• Elasticsearch offramp elastic: change the config value endpoints to nodes #732

New features

• Default to thin-lto for all builds (prior this was only done in docker)
• Automatically generate rpms and tarballs for releases.
• Update rust to 1.49.0
• Build deb packages
• Statically link openssl
• elastic sink now supports linked transports #715

Fixes

• rewrite string interpolation to fix #726