Skip to content

v2.0.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@jadams-r7 jadams-r7 released this 08 Jul 12:11
· 21 commits to main since this release
v2.0.0
eababba
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Broken Release Do Not Use

What's Changed

Fixes TXID metadata calculations:

  • New TXID Tracking Map: Introduced a new map with the structure commit uuid -> table name -> topic partition object -> txid for more granular and accurate transaction ID tracking.
  • Table Context Propagation: A new class was created to pass the table object into the coordinator's context. This provides the necessary table-level information for the new mapping. Dependent classes were updated to use this new context.
  • Centralized TXID Extraction: The responsibility for extracting transaction IDs has been moved to the Iceberg writer class. The extracted txids are now passed back within the written data results, streamlining the data flow.
  • Thread Safety: Implemented additional synchronization to ensure the new data structures are managed safely in a multi-threaded environment.
  • Empty File Filtering: Added logic to detect and filter out data files with zero records, preventing them from improperly influencing the txid metadata calculations.
  • Single Partition Valid Through: Added a partition check so on single partition topics the valid through no longer removes 1 keeping the data more up to date especially in low flowing topics improving test results and data lag as we are no longer unnecessarily holding onto the last transaction.

Full Changelog: rapid7:v1.3.2...rapid7:v2.0.0

Assets 4
Loading