ct: track l1 partition size #29482
Open
ct: track l1 partition size #29482
+841
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dotnwat
force-pushed
the
track-l1-partition-size
branch
from
053e0f9 to
5fae21e
Compare
Adds an interface to compute the summation of extent sizes for a partition that is managed by the simple metastore. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
dotnwat
force-pushed
the
track-l1-partition-size
branch
3 times, most recently
from
578bf5a to
ded128e
Compare
The size is tracked in the metadata_row_value, and the serde version is not incremented because we have not yet released this feature. Updates partition metadata for: * set_start_offset_db_update * add_objects_db_update * replace_objects_db_update Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
This relies on abstractions introduced at the metastore level in the previous commit which wired everything up, so this can't be squashed into the commit which altered the simple metastore itself. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
dotnwat
force-pushed
the
track-l1-partition-size
branch
from
ded128e to
7406981
Compare
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements partition size tracking for cloud topics in the L1 metastore, enabling clients to query the total byte size of a partition through a new GetSize RPC endpoint.
Changes:
- Adds a new
GetSizeRPC to the metastore service that returns the total byte size of a partition - Implements size tracking by maintaining a cumulative
sizefield in partition metadata that's updated incrementally during add, replace, and set_start_offset operations - Includes comprehensive test coverage for size calculations across various operations
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| proto/redpanda/core/admin/internal/cloud_topics/v1/metastore.proto | Defines the new GetSize RPC service and request/response messages |
| tests/rptest/clients/admin/proto/redpanda/core/admin/internal/cloud_topics/v1/metastore_pb2*.py* | Generated Python protobuf bindings for the GetSize RPC |
| tests/rptest/tests/cloud_topics/e2e_test.py | Adds end-to-end test verifying GetSize returns correct partition size |
| src/v/redpanda/admin/services/internal/metastore.{h,cc} | Implements admin API handler for GetSize RPC |
| src/v/cloud_topics/level_one/metastore/metastore.h | Adds GetSize method to the metastore interface |
| src/v/cloud_topics/level_one/metastore/simple_metastore.{h,cc} | Implements GetSize for the simple (in-memory) metastore |
| src/v/cloud_topics/level_one/metastore/replicated_metastore.{h,cc} | Implements GetSize for the replicated (production) metastore |
| src/v/cloud_topics/level_one/metastore/state.h | Adds calculate_size helper method for partition state |
| src/v/cloud_topics/level_one/metastore/lsm/values.h | Adds size field to metadata_row_value for persistent size tracking |
| src/v/cloud_topics/level_one/metastore/lsm/state_update.cc | Updates all state update operations to maintain the size field |
| src/v/cloud_topics/level_one/metastore/rpc_types.h | Defines internal RPC types for get_size requests and replies |
| src/v/cloud_topics/level_one/metastore/rpc.json | Registers the get_size RPC method |
| src/v/cloud_topics/level_one/metastore/service.{h,cc} | Implements RPC service handler for get_size |
| src/v/cloud_topics/level_one/metastore/leader_router.{h,cc} | Adds routing logic for get_size requests to the correct leader |
| src/v/cloud_topics/level_one/domain/domain_manager.h | Adds GetSize method to domain manager interface |
| src/v/cloud_topics/level_one/domain/simple_domain_manager.{h,cc} | Implements GetSize for simple domain manager |
| src/v/cloud_topics/level_one/domain/db_domain_manager.{h,cc} | Implements GetSize for database-backed domain manager |
| src/v/cloud_topics/level_one/metastore/tests/simple_metastore_test.cc | Unit tests for GetSize with various scenarios |
| src/v/cloud_topics/level_one/metastore/lsm/tests/state_update_test.cc | Unit tests verifying size tracking during state updates |
| src/v/cloud_topics/level_one/domain/tests/db_domain_manager_test.cc | Integration tests for GetSize at the domain manager level |
| tests/rptest/clients/admin/proto/redpanda/core/admin/internal/cloud_topics/v1/level_zero_gc_pb2*.py* | Updates to GC service protobuf bindings (documentation changes) |
rockwotj
reviewed
Feb 4, 2026
Contributor
rockwotj
left a comment
rockwotj
left a comment
There was a problem hiding this comment.
lgtm, just one comment on the test could be improved
| return False | ||
|
|
||
| wait_until( | ||
| condition=size_is_positive, |
Contributor
There was a problem hiding this comment.
maybe better to fold this into the retention tests and you can ensure retention book keeps this back to zero properly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Track the size of partitions in L1, and expose that through our internal admin interface.
This is part of a bigger project to support DescribeLogDirs Kafka API for cloud topics, which will require fusing together the metadata tracked in this PR with metadata from L0.
Backports Required
Release Notes