Fix high cardinality metrics by removing unbounded labels#2967
Open
fridrik01 wants to merge 1 commit intoblobreactor-prometheus2from
Open
Fix high cardinality metrics by removing unbounded labels#2967fridrik01 wants to merge 1 commit intoblobreactor-prometheus2from
fridrik01 wants to merge 1 commit intoblobreactor-prometheus2from
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## blobreactor-prometheus2 #2967 +/- ##
===========================================================
- Coverage 64.15% 64.11% -0.05%
===========================================================
Files 371 371
Lines 18597 18591 -6
===========================================================
- Hits 11931 11919 -12
- Misses 5759 5764 +5
- Partials 907 908 +1
🚀 New features to boost your workflow:
|
a6fbc96 to
9511120
Compare
784b633 to
78c4ed4
Compare
9511120 to
08758cf
Compare
08758cf to
c2d983d
Compare
c2d983d to
1591a48
Compare
4c886ad to
9a5c6eb
Compare
1591a48 to
232134c
Compare
9a5c6eb to
8666900
Compare
232134c to
b9d10c6
Compare
b9d10c6 to
6051072
Compare
99565e2 to
5fe4aab
Compare
6051072 to
a459d97
Compare
abi87
reviewed
Nov 25, 2025
| if err != nil { | ||
| s.logger.Error("Failed to read deposits", "error", err) | ||
| s.metrics.FailedToGetBlockLogs.With("block_num", blockNumStr).Add(1) | ||
| s.metrics.FailedToGetBlockLogs.Add(1) |
Contributor
There was a problem hiding this comment.
q: with this change, we should be able to get the time when the failure happens and retrieve the corresponding logs correct?
If so, I am all for this change
abi87
reviewed
Nov 25, 2025
| s.latestFcuReq.Store(&buildData.FCState) | ||
|
|
||
| s.metrics.markRebuildPayloadForRejectedBlockSuccess(nextBlkSlot) | ||
| s.metrics.markRebuildPayloadForRejectedBlockSuccess() |
Contributor
There was a problem hiding this comment.
just make sure we duly log where we are making the metrics less expressive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses a metrics issue by removing unbounded labels from Prometheus metrics that can cause cardinality explosion and degrade system performance in our monitoring systems.
I believe it should be fine to remove these labels as we should have information about them in our logs/loki.
Note: if any Grafana dashboards relied on these labels, they may need to be updated to reflect the new metric structure.