Release v0.12.0-rc.5

New features

• Add support for chunked requests and responses for the HTTP connectors
• Add the gbq connector for Google BigQuery

Fixes

• fix the use of args in the with part of a create insode of a flow.
• fix silent swallowing of unknown fields in connector definition.

Breaking Changes

• the - is no longer a valid part of identifiers.
• binaries now use _ to sepoerate type names as - is no longer a identifier.

Release v0.12.0-rc.4

New features

• Add support for chunked requests and responses for the HTTP connectors
• Add the gbq connector for Google BigQuery

Fixes

• fix the use of args in the with part of a create insode of a flow.
• fix silent swallowing of unknown fields in connector definition.

Breaking Changes

• the - is no longer a valid part of identifiers.
• binaries now use _ to sepoerate type names as - is no longer a identifier.

Release v0.12.0-rc.2

New features

• Add support for chunked requests and responses for the HTTP connectors
• Add the gbq connector for Google BigQuery

Fixes

• fix the use of args in the with part of a create insode of a flow.
• fix silent swallowing of unknown fields in connector definition.

Breaking Changes

• the - is no longer a valid part of identifiers.
• binaries now use _ to sepoerate type names as - is no longer a identifier.

Release v0.12.0-rc.1

0.12.0-rc.1

New featues

• Add support for tuple patterns inside of record patterns.
• Refactor visitors to seperate walker and visitor and visit all nodes.
• Add support for modular subqueries in Trickle
• Add check_topic_metadata configuration flag to kafka source to bypass topic metadata fetch
• port json! improvements to literal!
• print names of failing tests
• Add check_topic_metadata configuration flag to kafka source to bypass topic metadata fetch
• port json! improvements to literal!
• Add support for troy deployment language. The language adds the pipeline, connector, flow, links, connect, to, deploy and config reserved keywords. These are now reserved in the scripting and query languages and must now be escaped when they appear in event data. This is a breaking change in the query and script language dialects.
• Add support for FLOAT4 and FLOAT8 serialization/deserialization to postgres connectors
• Add std::path::try_default fn
• Experimental ARM support
• Add default => {...} and default "key" => "value" to patch
• Add zstd pre- and post-processors #1100
• Remove rental from Event #1031 #1037
• Put event raw payload into Arc to improve cloning perf
• Remove rental from the entire runtime #1037
• Restructure operators and select to avoid transmutation #1024
• Fix tremor-cli to use it's own binary when possible #1096
• Restructure functions to avoid transmutation #1028
• Update blackhole to print events / s #1129
• Improve soundness and documentation of SRS code.
• Add support for concatenating arrays #1113
• Allow gcp headers to be included in rest offramp with auth: gcp
• Add env source #1136
• Remove need for emit_emtpy_window by automatic eviction for groups where no window holds data
• Ensure all windows of a group are alligned and can not go out of sync
• Box apropriately in the rest sink
• replace macros with functions in gcp code
• Add -q flag and clarify -v flag for unit tests
• Add win::cardinality function
• Backpressure now allows lossless (no discard) circuit breaker behaviour #1119
• Allow functions, constants and expressions as roots for path lookups
• Add the unix-socket source
• Enabale automatic benchmakrs
• Add the csv codec
• Add the permissions option for setting the file mode for the unix-socket source
• Tests can be run without their suite. #1238
• Add the std::size module to convert sizes
• Add custom function calls to constant folding
• Integration test names are added as a tag so they can be run by name
• integration tests low log stdout/stderr for before and after

Fixes

• Make otel severity_number optional: #1248
• Don't allow duplicate stream names: #1212
• Fix memory safety issue when using merge or patch with state as target and reassigning the resulting value to state #1217
• Fix delayed event re-execution in case of errors in a branched pipeline
• Skip instead of fail EQC on out of repo PRs
• Ensure patch keys are strings to move runtime errors into the compiletime
• Fix issue with the token " being presented as a tick in errors
• Fix heredoc_start and heredoc_end showing up in error messages instead of """
• Fix some errors in otel pb <-> json translation
• Fix windowed queries emitting events with null metadata on tick
• Fix sorting for artefacts
• Fix issue where the test framework would generate reports without being asked for it #1072
• Remove the need for eviction_period for time based windows
• Remove dead code and unneeded allows in otel and gcp code
• Fix test and suite names not being printed
• Fix badly nested structure in unit tests
• Fix bug in the unit testing framework that would ignore all tags
• Fix the illogical structure of suites that required a doubly nested record
• Fix -v flag
• Fix issue with double counting of unit test stats
• Fix issue with wrong script snippets being shown for unit tests
• Fix argument order in test cases
• Fix GCS go-auth token refresh
• Fix create script syntax for aliased scripts with overridden params
• Add benchmark names to benchmark tags
• Kafka onramp: Remove failing metadata fetch in order to verify topic existance. Instead detect subscription errors and stop the onramp in that case.
• Unix offramp: Add the missing StartStream message
• tremor-script: Add more details about Unicode in the documentation of the string module
• Fix hdr and dds aggregation function losing events when aggregating > 8192 events
• Ensure merge can only happen on objects

Breaking Changes

• changed naming for record object to avoid keywords like select and merge. New names are record.extract and record.combine.
• command separators are now unified, both patch, match and for now use ; the same way the rest of the language does
• in all definitional statements args now specifies interface arguments that are overwritable in the correspanding create statement, while with specifies non-overwritable configuration in both define and create statements - this unifies the use of with and args between trickle and troy
• file connector no longer splits by lines - it now requires a preconnector
• define for both troy and trickle now follow the same principle of define <type> <alias> from <source>
• wal is no longer an operator but a connector
• for the elastic connector indexes have not to be set on the batch not the individual event so one batch can only be to a single index.
• metronome interval is now in nanoseconds (as all other timings)
• Most connectors require a specified codec now instead of using JSON as a default
• merge no longer treats null in the spec as a delete option but rather as a normal value
• Combine all compression and decompression pre/postprocessors.

v0.11.12

0.11.12

Fixes

• Unix onramp: Add the missing StartStream message to handle multiple connections correctly

v0.11.11

Fix issue with the kafka connector that broke delivery guarantees introduced during the 0.11.7 update of librdkafka

v0.11.10

0.11.10

Fixes

• Fix windowed queries emitting events with null metadata on tick

v0.11.9

0.11.9

Fixes

• Kafka onramp: Remove failing metadata fetch in order to verify topic existence. Instead detect subscription errors and stop the onramp in that case.

Details

The Kafka onramp used to fail non-reliably during startup when attempting to verify topic existence by fetching the metadata for all given topics. This was error prone. Additionally we hard-coded a timeout that was way too tight, especially for big clusters. We removed this check and instead error properly if we find out that a subscription did not work. This only works asynchronously, so the only way is to stop the source. The Tremor logs will show the error. Example:

2021-10-20T13:16:52.549478+02:00 INFO tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscribing to: ["test", "snot"]
2021-10-20T13:16:52.549752+02:00 INFO tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscription initiated...
2021-10-20T13:16:52.550037+02:00 INFO tremor_runtime::onramp - Onramp tremor://localhost/onramp/datain/simplees01/out started.
2021-10-20T13:16:52.550816+02:00 INFO tremor_runtime::source - [Source::tremor://localhost/onramp/datain/simplees01/out] Connecting tremor://localhost/pipeline/system::metrics/system/in as metrics pipeline.
2021-10-20T13:16:52.551138+02:00 INFO tremor_runtime::pipeline - [Pipeline::tremor://localhost/pipeline/system::passthrough/simplees01] Connecting tremor://localhost/onramp/datain/simplees01/out to 'in'
2021-10-20T13:16:52.551325+02:00 INFO tremor_runtime::system - Binding binding tremor://localhost/binding/default/simplees01
2021-10-20T13:16:52.567448+02:00 ERROR tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscription failed: UnknownTopicOrPartition (Broker: Unknown topic or partition). Stopping.
2021-10-20T13:16:52.567547+02:00 WARN tremor_runtime::source - [Source::tremor://localhost/onramp/datain/simplees01/out] Disconnected.

v0.11.8

0.11.8

Fixes

• Fix hdr and dds aggregation function losing events when aggregating > 8192 events: #1246
• Make otel severity_number optional: #1248

New Features

• Add the unix-socket onramp: #1236 By @Agares

v0.11.7 - Upgrade librdkafka to 1.6.1

We crafted this release in order to mitigate a possible bug in librdkafka: confluentinc/librdkafka#2933

Fixes

• Upgrade dependency on librdkafka to 1.6.1 #1228.