Skip to content

Latest commit

 

History

History
27 lines (20 loc) · 1.38 KB

_hypercore-direct-compress.md

File metadata and controls

27 lines (20 loc) · 1.38 KB

When you set timescaledb.enable_direct_compress_copy your data gets compressed in memory during ingestion with COPY statements. By writing the compressed batches immediately in the columnstore, the IO footprint is significantly lower. Also, the columnstore policy you set is less important, INSERT already produces compressed chunks.

Please note that this feature is a tech preview and not production-ready. Using this feature could lead to regressed query performance and/or storage ratio, if the ingested batches are not correctly ordered or are of too high cardinality.

To enable in-memory data compression during ingestion:

SET timescaledb.enable_direct_compress_copy=on;

Important facts

  • High cardinality use cases do not produce good batches and lead to degreaded query performance.
  • The columnstore is optimized to store 1000 records per batch, which is the optimal format for ingestion per segment by.
  • WAL records are written for the compressed batches rather than the individual tuples.
  • Currently only COPY is support, INSERT will eventually follow.
  • Best results are achieved for batch ingestion with 1000 records or more, upper boundary is 10.000 records.
  • Continous Aggregates are not supported at the moment.