Failure: altinitystable-25-8-antalya-x86 - /iceberg/iceberg cache

[CarlosFelipeOR]

Version: 25.8.12.20747.altinityantalya

Description

Several tests under /iceberg/iceberg cache are failing on Antalya 25.8 (x86).
The failures started in Antalya 25.3.3.20183 due to a regression in how Parquet metadata cache ProfileEvents are emitted.

While the same test passes in earlier Antalya 25.3 builds, starting from 25.3.3.20183 the ParquetMetaDataCacheHits ProfileEvent is no longer present in system.query_log during Iceberg database engine queries, even when input_format_parquet_use_metadata_cache = 1 and Parquet files are accessed.

This behavior breaks the expected SRS contract for Parquet metadata caching and causes the Iceberg cache suite to fail consistently in newer Antalya releases.

Steps to Reproduce

Option 1: Reproduce via automation

The failing test can be reproduced by running the Iceberg regression suite with the following command:

python3 -u iceberg/regression.py \
  --only "/iceberg/iceberg cache/rest catalog/iceberg database engine/cache/*" \
  --local \
  --clickhouse docker://altinity/clickhouse-server:25.8.12.20747.altinityantalya \
  --clickhouse-version 25.8.12.20747 \
  --pause-on-fail "/iceberg/iceberg cache/rest catalog/iceberg database engine/cache/"

The execution will pause immediately after the failure, allowing direct inspection of the environment and investigation using system.query_log.

Option 2: Manual reproduction

The issue can also be reproduced manually by following the step-by-step instructions described in the comment below (Iceberg + Parquet metadata cache setup and query execution).

Analysis

This failure is caused by a regression in Parquet metadata cache accounting for the Iceberg database engine path.

The test validates the SRS requirement that, when input_format_parquet_use_metadata_cache = 1, ClickHouse must:

• cache Parquet file metadata when querying data from remote object storage, and
• emit the ParquetMetaDataCacheHits ProfileEvent in system.query_log.

The investigation shows a clear behavioral change between Antalya versions:

• In Antalya 25.3.3.20143, the Iceberg database engine consistently emits ParquetMetaDataCacheHits > 0 for the warm runs of the same query.
• Starting from Antalya 25.3.3.20183 and continuing in Antalya 25.8, the same Iceberg queries no longer emit ParquetMetaDataCacheHits at all (the event does not appear even as a key in ProfileEvents), despite:
  • input_format_parquet_use_metadata_cache being enabled,
  • Parquet files being accessed and decoded, and
  • multiple warm executions of the same query.

This indicates that the Iceberg database engine path in newer Antalya versions either bypasses the Parquet metadata cache or no longer propagates its cache ProfileEvents to system.query_log.

As a result, the test failure is deterministic and reflects a real regression in Parquet metadata cache behavior or telemetry, not a test instability.

Failing tests

✘ [ Fail ] '/iceberg/iceberg cache/rest catalog/iceberg database engine/cache' (35s 112ms)
✘ [ Fail ] '/iceberg/iceberg cache/rest catalog/iceberg database engine' (35s 114ms)
✘ [ Fail ] '/iceberg/iceberg cache/rest catalog/icebergS3 table function/cache' (43s 44ms)
✘ [ Fail ] '/iceberg/iceberg cache/rest catalog/icebergS3 table function' (43s 45ms)
✘ [ Fail ] '/iceberg/iceberg cache/rest catalog' (1m 22s)
✘ [ Fail ] '/iceberg/iceberg cache/glue catalog/iceberg database engine/cache' (32s 488ms)
✘ [ Fail ] '/iceberg/iceberg cache/glue catalog/iceberg database engine' (32s 489ms)
✘ [ Fail ] '/iceberg/iceberg cache/glue catalog/icebergS3 table function/cache' (44s 566ms)
✘ [ Fail ] '/iceberg/iceberg cache/glue catalog/icebergS3 table function' (44s 567ms)
✘ [ Fail ] '/iceberg/iceberg cache/glue catalog' (1m 20s)
✘ [ Fail ] '/iceberg/iceberg cache' (2m 42s)

---

CI Run

https://github.com/Altinity/clickhouse-regression/actions/runs/20752126541/job/59639226319

Artifacts and Reports

https://altinity-internal-test-reports.s3.amazonaws.com/index.html#clickhouse/25.8.12.20747/20752126541/testflows/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/iceberg1/iceberg/

[CarlosFelipeOR]

Update

This test has been failing for a long time only in altinity/clickhouse-regression.

In altinity/ClickHouse, it was not failing because it was not being executed at all due to a syntax error in the run command:
the /* wildcard was missing in --only "/iceberg/iceberg cache".

As a result, the /iceberg/iceberg cache suite was silently skipped in altinity/ClickHouse, which masked the issue there.

[CarlosFelipeOR]

Update / Analysis

After a deeper investigation, we confirmed this is not an intermittent failure nor a test bug, but a behavior change / regression introduced in Antalya 25.3 and still present in later versions (including Antalya 25.8).

Historical behavior (passes in older versions)

By querying our historical test database and re-running the same automation against multiple ClickHouse versions, we confirmed that this test used to pass in earlier Antalya 25.3 builds.

Specifically:

• PASS: 25.3.3.20143 (released on Jun 13, 2025, 10:10 AM)
• FAIL: 25.3.3.20183 (released on Jul 10, 2025, 1:54 PM)

This allowed us to narrow down the regression window precisely to the range between these two builds.

Version where the regression starts

The first version where the failure consistently appears is:

• 25.3.3.20183.altinityantalya

Release notes for this version:
https://github.com/Altinity/ClickHouse/releases/tag/v25.3.3.20183.altinityantalya

Relevant changes in the release notes

The release notes for 25.3.3.20183.altinityantalya include several Iceberg-related changes that could directly impact this test behavior, most notably:

• Changes to the default behavior of Iceberg partition pruning, which may alter when and how Parquet metadata is accessed during query execution.
• Improvements and refactoring in Iceberg metadata handling, including changes around metadata files and object storage interaction.
• Updates related to Iceberg object storage paths and metadata access, which can affect cache keys and the code paths responsible for emitting cache-related ProfileEvents.

Given that this test relies on observing ParquetMetaDataCacheHits during Iceberg database engine queries, any change that reduces or bypasses Parquet metadata access—or routes it through a different caching layer—can explain why the ProfileEvent stopped being emitted starting from this release.

[CarlosFelipeOR]

Steps to Reproduce

Prerequisites

• ClickHouse Antalya build:
  • A version >= 25.3.3.20183.altinityantalya (e.g. 25.8.x) to reproduce the failure
  • A version 25.3.3.20143.altinityantalya to observe the passing behavior
• Access to an S3-compatible object storage (e.g. MinIO)

Environment setup

1. Start an S3-compatible object storage (e.g. MinIO).
2. Create an Iceberg catalog pointing to the object storage.
3. Create:
   • an Iceberg namespace
   • an Iceberg table stored in Parquet format, with at least a few data files
4. Create a ClickHouse database using the Iceberg database engine pointing to the same warehouse location.

---

Reproduction steps

1. Ensure Parquet metadata cache is enabled:

   SET input_format_parquet_use_metadata_cache = 1;

2. Execute a query against the Iceberg database engine that does not return rows, but still evaluates metadata (example):

   SELECT *
   FROM iceberg_database.namespace.table
   WHERE date_col > '3030-01-01'
   SETTINGS log_comment = 'repro_parquet_metadata_cache';

3. Repeat the same query multiple times (e.g. 50–100 executions) to allow cache reuse.

4. Inspect the query log:

   SELECT
       count() AS total_rows,
       sum(mapContains(ProfileEvents, 'ParquetMetaDataCacheHits')) AS rows_with_parquet_hits_key,
       sum(ProfileEvents['ParquetMetaDataCacheHits']) AS parquet_hits_sum,
       sum(ProfileEvents['IcebergMetadataFilesCacheHits']) AS iceberg_hits_sum
   FROM system.query_log
   WHERE type = 'QueryFinish'
     AND log_comment = 'repro_parquet_metadata_cache';

---

Expected result (per SRS)

• rows_with_parquet_hits_key > 0
• parquet_hits_sum > 0

This indicates that Parquet metadata was reused from cache and that
ParquetMetaDataCacheHits was properly tracked in system.query_log.

Actual result

• Antalya 25.3.3.20143 (PASS):

  • ParquetMetaDataCacheHits is present and > 0 for each warm execution.

• Antalya 25.3.3.20183 and newer (FAIL):

  • ParquetMetaDataCacheHits is not present in ProfileEvents
  • The key does not appear even in mapKeys(ProfileEvents)
  • This occurs even when Parquet files are read and decoded.

---

Notes

• The issue reproduces both when:

  • queries return no rows (full pruning), and
  • queries are modified to force Parquet reads (read_rows > 0)

• This indicates the problem is not related to query shape or pruning,
  but to a change in the Parquet metadata caching or telemetry path.