Skip to content

Commit 5117427

Browse files
authored
Document common OpMon use cases (#883)
* Document common OpMon use cases * Fix experimenter links
1 parent 0233490 commit 5117427

File tree

3 files changed

+20
-4
lines changed

3 files changed

+20
-4
lines changed

src/concepts/metric_hub.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Metric Hub is a [repository](https://github.com/mozilla/metric-hub) that contains metric, data source and segment definitions that have been
44
reviewed and can be seen as the source of truth.
5-
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/deep-dives/jetstream/overview) and [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon).
5+
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview) and [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon).
66

77
Metric Hub is a "thin" metric layer that lies between the Data Warehouse and consumers, such as analysis tools.
88
The Data Warehouse contains the datasets and tables referenced in the metric definitions. Analysis tools can reference metrics that are defined in metric-hub.
@@ -256,7 +256,7 @@ Segment SQL snippets need to be boolean expressions to be valid.
256256

257257
All the definitions are automatically available in some of our tooling:
258258

259-
- [Jetstream](https://experimenter.info/deep-dives/jetstream/overview) - used for analyzing experiments
259+
- [Jetstream](https://experimenter.info/data-analysis/jetstream/overview) - used for analyzing experiments
260260
- [mozanalysis](https://github.com/mozilla/mozanalysis) - a Python library which standardizes how experiment data is analyzed at Mozilla
261261
- [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html) - a tool for monitoring operational metrics
262262
- [bigquery-etl](https://github.com/mozilla/bigquery-etl) - used for writing ETL queries

src/cookbooks/operational_monitoring.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Access to the Looker Operational Monitoring dashboards is currently limited to M
1717
## Configuring a Operational Monitoring project
1818

1919
To add or update a project configuration, open a pull request against the `opmon/` directory in [metric-hub](https://github.com/mozilla/metric-hub/tree/main/opmon).
20-
Consider using and adding metric definitions to the [metric-hub](https://github.com/mozilla/metric-hub) that should be available for use across other tools, such as [Jetstream](https://experimenter.info/deep-dives/jetstream/overview), as well.
20+
Consider using and adding metric definitions to the [metric-hub](https://github.com/mozilla/metric-hub) that should be available for use across other tools, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview), as well.
2121

2222
CI checks will validate the columns, data sources, and SQL syntax. Once CI completes, the pull request gets automatically approved and can be merged. Results for the new project will be available within the next 24 hours.
2323

@@ -460,6 +460,22 @@ The table `projects_v1` in `operational_monitoring_derived` contains metadata ab
460460

461461
OpMon is updated nightly by telemetry-airflow. It is invoked by the [operational_monitoring DAG](https://github.com/mozilla/telemetry-airflow/blob/main/dags/operational_monitoring.py).
462462

463+
## Common Use Cases
464+
465+
- **Rollout Monitoring:** Monitor rollouts to detect regressions before they affect the entire population. Compares enabled vs disabled branches for search metrics, performance, and error rates. The rollout monitoring dashboards get created automatically.
466+
- **Build-over-Build Performance:** Detect performance regressions between consecutive builds on pre-release channels (Beta, Nightly). Tracks page load, FCP, input latency, JS execution, and memory metrics.
467+
- **Product Health Dashboards:** Continuous monitoring of stability metrics without end dates: crash rates, hangs, exceptions, success/failure rates.
468+
- **DAU/MAU Tracking and other Business Metrics:** Aggregate-level monitoring of daily/monthly active users, often with KPI forecasting to detect retention issues. Track revenue-impacting metrics like search monetization, subscription engagement, and customer satisfaction.
469+
- **Experiments & A/B Tests:** As an addition to Experimenter, OpMon can be used to track experiment-specific success criteria and side effects for earlier signals.
470+
- **Background Processes:** Monitor Firefox's background services (Background Update, Default Browser Agent). Tracks success rates, exception volumes, and timing metrics, often segmented by OS version.
471+
472+
## When to use OpMon?
473+
474+
- OpMon → Continuous monitoring, rollouts, build-over-build comparisons
475+
- Experimenter → A/B tests requiring statistical significance testing
476+
- GLAM → Ad-hoc histogram exploration
477+
- Custom SQL → Complex queries beyond OpMon's aggregation model
478+
463479
## Experiments vs OpMon
464480

465481
The requirements for Operational Monitoring are related to, but mostly distinct from those for experiments:

src/datasets/jetstream.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,4 +211,4 @@ by interacting with the [`jetstream-config`] repository.
211211

212212
## Documentation
213213

214-
Additional documentation about Jetstream can be found in the [Jetstream Experimenter Docs](https://experimenter.info/deep-dives/jetstream/overview).
214+
Additional documentation about Jetstream can be found in the [Jetstream Experimenter Docs](https://experimenter.info/data-analysis/jetstream/overview).

0 commit comments

Comments
 (0)