Rust component metrics #8819
Merged
Rust component metrics #8819
+448
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bendk
commented
Feb 5, 2026
| CROSS JOIN UNNEST(metrics.{{ metric.table }}.{{ category }}_{{ metric.name }}.values) as values | ||
| -- This generates multiple rows based on the `value` field. This is needed to make the `APPROX_QUANTILES` | ||
| -- weigh `value.key` correctly. | ||
| CROSS JOIN UNNEST(GENERATE_ARRAY(1, `values`.value)) |
Contributor
Author
There was a problem hiding this comment.
Is this the best way to get percentiles for a distribution metric? I tried to find a UDF for this, but couldn't.
bendk
commented
Feb 5, 2026
scholtzan
reviewed
Feb 5, 2026
scholtzan
reviewed
Feb 6, 2026
bendk
force-pushed
the
push-ltmxmvkrqvsq
branch
2 times, most recently
from
4c0d58d to
45b1c7c
Compare
Created the `rust_component_metrics` dataset, which contains data for shared Rust components that ship on Desktop, iOS, and/or Android. See https://github.com/mozilla/application-services/ for examples of these components. Added SQL generators to create derived datasets that aggregate Glean metrics. The aggregate tables will be used to create dashboards for teams that own the Rust components. There's already some code in the application-services repo to generate these dashboards, however the queries are running slowly and often time out. Currently supported metrics are counters, distributions, labeled distributions and events. SQL generators were used in order to make it easy for teams to add metrics for their components in the future. We can tell them to update `sql_generators/rust_component_metrics/__init__.py` and open a PR. I could almost have just used the GLAM ETL tables, but we need some extra capabilities: * firefox-ios support (mozilla/glam#1830) * Aggregation by submission date (mozilla/glam#1073)
Contributor
Author
|
Fixed the YAML formatting and the partition date field name. Hopefully this makes CI green. |
scholtzan
reviewed
Feb 9, 2026
| metrics=[ | ||
| LabeledDistribution("ingest_download_time", DistributionType.timing), | ||
| LabeledDistribution("ingest_time", DistributionType.timing), | ||
| LabeledDistribution("ingest_query_time", DistributionType.timing), |
Collaborator
There was a problem hiding this comment.
The dry-run CI task is failing because it looks like this the suggest_ingest_query_time doesn't actually exist in the source table.
Collaborator
There was a problem hiding this comment.
This is the full dryrun error for the query:
sql/*****************************/rust_component_derived_moz_fx_data_shared_prod_2707a00cf2b29f0afd4e378935d3378c99f6599e_761c7b67/ingest_query_time_v1/query.sql ERROR
[{'code': 400, 'errors': [{'message': 'Field name suggest_ingest_query_time does not exist in STRUCT<network_http3_complete_load ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, network_http3_first_sent_to_last_received ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, network_http3_open_to_first_received ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, ...>; Did you mean suggest_ingest_time? at [21:50]', 'domain': 'global', 'reason': 'invalidQuery', 'location': 'q', 'locationType': 'parameter'}], 'response': {'headers': {'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 09 Feb 2026 18:29:08 GMT', 'server': 'ESF', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'x-content-type-options': 'nosniff', 'x-frame-options': 'SAMEORIGIN', 'x-xss-protection': '0'}}, 'message': 'Field name suggest_ingest_query_time does not exist in STRUCT<network_http3_complete_load ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, network_http3_first_sent_to_last_received ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, network_http3_open_to_first_received ARRAY<STRUCT<key STRING, value STRUCT<bucket_count INT64, count INT64, histogram_type STRING, ...>>>, ...>; Did you mean suggest_ingest_time? at [21:50]'}]
Contributor
Author
There was a problem hiding this comment.
Sorry about that typo. I changed it to query_time, which should fix it.
scholtzan
reviewed
Feb 9, 2026
scholtzan
approved these changes
Feb 9, 2026
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 9, 2026
* Rust component metrics Created the `rust_component_metrics` dataset, which contains data for shared Rust components that ship on Desktop, iOS, and/or Android. See https://github.com/mozilla/application-services/ for examples of these components. Added SQL generators to create derived datasets that aggregate Glean metrics. The aggregate tables will be used to create dashboards for teams that own the Rust components. There's already some code in the application-services repo to generate these dashboards, however the queries are running slowly and often time out. Currently supported metrics are counters, distributions, labeled distributions and events. SQL generators were used in order to make it easy for teams to add metrics for their components in the future. We can tell them to update `sql_generators/rust_component_metrics/__init__.py` and open a PR. I could almost have just used the GLAM ETL tables, but we need some extra capabilities: * firefox-ios support (mozilla/glam#1830) * Aggregation by submission date (mozilla/glam#1073) * Apply suggestion from @scholtzan --------- Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created the
rust_component_metricsdataset, which contains data for shared Rust components that ship on Desktop, iOS, and/or Android. See https://github.com/mozilla/application-services/ for examples of these components.Added SQL generators to create derived datasets that aggregate Glean metrics. The aggregate tables will be used to create dashboards for teams that own the Rust components. There's already some code in the application-services repo to generate these dashboards, however the queries are running slowly and often time out. Currently supported metrics are counters, distributions, labeled distributions and events.
SQL generators were used in order to make it easy for teams to add metrics for their components in the future. We can tell them to update
sql_generators/rust_component_metrics/__init__.pyand open a PR.I could almost have just used the GLAM ETL tables, but we need some extra capabilities:
Description
Related Tickets & Documents
Reviewer, please follow this checklist