docs: Iterate on the OTel Engine Documentation

docs/sources/_index.md

@@ -11,6 +11,9 @@ cascade:
   BEYLA_VERSION: v2.8.5
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
+  FULL_OTEL_ENGINE: Alloy OpenTelemetry Engine
+  OTEL_ENGINE: OTel Engine
+  DEFAULT_ENGINE: Default Engine
 hero:
   title: Grafana Alloy
   level: 1

docs/sources/_index.md.t

@@ -11,6 +11,9 @@ cascade:
   BEYLA_VERSION: v2.8.5
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
+  FULL_OTEL_ENGINE: Alloy OpenTelemetry Engine
+  OTEL_ENGINE: OTel Engine
+  DEFAULT_ENGINE: Default Engine
 hero:
   title: Grafana Alloy
   level: 1

docs/sources/opentelemetry/_index.md

@@ -0,0 +1,149 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/opentelemetry/
+description: Grafana Alloy is a flexible, high performance, vendor-neutral distribution of the OpenTelemetry Collector
+menuTitle: OpenTelemetry Engine
+title: The OpenTelemetry Engine
+_build:
+  list: false
+noindex: true
+weight: 10
+---
+
+# The {{% param "FULL_OTEL_ENGINE" %}}
+
+{{< param "FULL_PRODUCT_NAME" >}} combines the Prometheus-native, production-grade collection features of {{< param "PRODUCT_NAME" >}} with the broad ecosystem and standards of OpenTelemetry.
+The {{< param "FULL_OTEL_ENGINE" >}} is a bundled OpenTelemetry Collector distribution embedded within {{< param "PRODUCT_NAME" >}} that lets you run {{< param "PRODUCT_NAME" >}} as a fully compatible OTel Collector while retaining access to all {{< param "PRODUCT_NAME" >}} features and integrations.
+
+{{< docs/shared lookup="stability/experimental_otel.md" source="alloy" version="<ALLOY_VERSION>" >}}
+
+## Terminology
+
+{{< param "PRODUCT_NAME" >}} supports two runtime engines and an extension:
+
+- **{{< param "DEFAULT_ENGINE" >}}**: The default {{< param "PRODUCT_NAME" >}} runtime and [configuration syntax](../get-started/syntax/).
+  This remains the default, stable experience with [backward compatibility](../introduction/backward-compatibility/) guarantees for {{< param "PRODUCT_NAME" >}} users.
+
+- **{{< param "OTEL_ENGINE" >}}**: The standard OpenTelemetry Collector runtime embedded within {{< param "PRODUCT_NAME" >}}.
+  It uses [upstream collector YAML configuration](https://opentelemetry.io/docs/collector/configuration/) for pipelines and components.
+
+- **{{< param "PRODUCT_NAME" >}} Engine extension**: An OpenTelemetry Collector extension that allows you to run both the {{< param "DEFAULT_ENGINE" >}} and the {{< param "OTEL_ENGINE" >}} in parallel.
+
+## Included components
+
+The {{< param "OTEL_ENGINE" >}} bundle includes:
+
+- Standard components from the OpenTelemetry Collector core
+- A curated selection of components from contributor repositories
+- The `alloyengine` extension
+
+The following sections list all included components:
+
+{{< collapse title="Extensions" >}}
+
+- [alloyengine](https://github.com/grafana/alloy/tree/main/extension/alloyengine)
+- [basicauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension)
+- [bearertokenauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/bearertokenauthextension)
+- [headerssetter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextension)
+- [healthcheck](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextension)
+- [jaegerremotesampling](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/jaegerremotesampling)
+- [oauth2clientauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/oauth2clientauthextension)
+- [pprof](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/pprofextension)
+- [sigv4auth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/sigv4authextension)
+- [filestorage](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage)
+- [zpages](https://github.com/open-telemetry/opentelemetry-collector/tree/main/extension/zpagesextension)
+
+{{< /collapse >}}
+
+{{< collapse title="Configuration Providers" >}}
+
+- [env](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/envprovider)
+- [file](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/fileprovider)
+- [http](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/httpprovider)
+- [https](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/httpsprovider)
+- [yaml](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/yamlprovider)
+
+{{< /collapse >}}
+
+{{< collapse title="Receivers" >}}
+
+- [awscloudwatch](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver)
+- [awsecscontainermetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsecscontainermetricsreceiver)
+- [awss3](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver)
+- [cloudflare](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/cloudflarereceiver)
+- [datadog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver)
+- [faro](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/faroreceiver)
+- [filelog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver)
+- [filestats](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filestatsreceiver)
+- [fluentforward](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver)
+- [googlecloudpubsub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/googlecloudpubsubreceiver)
+- [hostmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver)
+- [influxdb](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/influxdbreceiver)
+- [jaeger](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver)
+- [kafka](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkareceiver)
+- [solace](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/solacereceiver)
+- [splunkhec](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver)
+- [syslog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/syslogreceiver)
+- [tcplog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver)
+- [vcenter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/vcenterreceiver)
+- [zipkin](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver)
+- [otlp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver)
+{{< /collapse >}}
+
+{{< collapse title="Connectors" >}}
+- [count](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector)
+- [grafanacloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/grafanacloudconnector)
+- [servicegraph](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector)
+- [spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector)
+- [forward](https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector)
+
+{{< /collapse >}}
+
+{{< collapse title="Processors" >}}
+
+- [attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor)
+- [cumulativetodelta](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor)
+- [deltatocumulative](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor)
+- [filter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor)
+- [groupbyattrs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbyattrsprocessor)
+- [interval](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor)
+- [k8sattributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor)
+- [metricstarttime](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstarttimeprocessor)
+- [probabilisticsampler](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor)
+- [resource](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor)
+- [resourcedetection](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor)
+- [span](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanprocessor)
+- [tailsampling](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
+- [transform](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor)
+- [batch](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor)
+- [memorylimiter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor)
+
+{{< /collapse >}}
+
+{{< collapse title="Exporters" >}}
+
+- [awss3](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awss3exporter)
+- [faro](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/faroexporter)
+- [file](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/fileexporter)
+- [googlecloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter)
+- [googlecloudpubsub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudpubsubexporter)
+- [kafka](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter)
+- [loadbalancing](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter)
+- [prometheus](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter)
+- [prometheusremotewrite](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter)
+- [splunkhec](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter)
+- [syslog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/syslogexporter)
+- [zipkin](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/zipkinexporter)
+- [debug](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter)
+- [nop](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/nopexporter)
+- [otlp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter)
+- [otlphttp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter)
+
+{{< /collapse >}}
+
+To view the full list of components and their versions, refer to the [OpenTelemetry Collector Builder manifest](https://github.com/grafana/alloy/blob/main/collector/builder-config.yaml).
+
+## Next steps
+
+- [Get started](./get-started/) with the {{< param "OTEL_ENGINE" >}}
+- [OTel CLI](../reference/cli/otel/) reference documentation
+

docs/sources/opentelemetry/get-started.md

@@ -0,0 +1,170 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/opentelemetry/get-started
+description: Get started with the Alloy OpenTelemetry Engine
+menuTitle: Get Started
+title: Get Started with the Alloy OpenTelemetry Engine
+_build:
+  list: false
+noindex: true
+weight: 10
+---
+
+# Get Started with the {{% param "FULL_OTEL_ENGINE" %}}
+
+You can run the {{< param "OTEL_ENGINE" >}} using the CLI, Helm chart, or service installation.
+
+## Prerequisites
+
+There are no additional prerequisites.
+The tools needed to run the {{< param "OTEL_ENGINE" >}} are shipped within {{< param "PRODUCT_NAME" >}}.
+
+Before you start, validate your OpenTelemetry YAML configuration with the `validate` command:
+
+```bash
+./build/alloy otel validate --config=<CONFIG_FILE>
+```
+
+## Run with the CLI
+
+The {{< param "OTEL_ENGINE" >}} is available under the {{< param "PRODUCT_NAME" >}} `otel` command.
+The CLI is the easiest way to experiment locally or on a single host.
+Refer to the [OTel CLI](../../reference/cli/otel/) documentation for more information.
+
+The following example configuration file accepts telemetry over [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/) and sends it to the Grafana Cloud OTLP gateway.
+You can adapt the endpoint and credentials for your region and tenant.
+
+```yaml
+extensions:
+  basicauth/grafana_cloud:
+    client_auth:
+      username: <INSTANCE_ID>
+      password: <API_TOKEN>
+
+receivers:
+  otlp:
+    protocols:
+      grpc: {}
+      http: {}
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 512
+
+exporters:
+  otlphttp/grafana_cloud:
+    endpoint: <GRAFANA_URL>
+    auth:
+      authenticator: basicauth/grafana_cloud
+
+service:
+  extensions: [basicauth/grafana_cloud]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlphttp/grafana_cloud]
+```
+
+Replace the following:
+
+- _`<INSTANCE_ID>`_: Your Grafana Cloud instance ID.
+- _`<API_TOKEN>`_: Your Grafana Cloud API token.
+- _`<GRAFANA_URL>`_: Your Grafana Cloud OTLP endpoint URL.
+
+For more information about where to find these values, refer to [Send data using OpenTelemetry Protocol](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/).
+
+To start the {{< param "OTEL_ENGINE" >}}, run the following command:
+
+```shell
+alloy otel --config=<CONFIG_FILE> [<FLAGS> ...]
+```
+
+{{< param "PRODUCT_NAME" >}} then accepts incoming OTLP data on `0.0.0.0:4317` for gRPC and `0.0.0.0:4318` for HTTP requests.
+Metrics are also available on the default collector port and endpoint at `0.0.0.0:8888/metrics`.
+Since the {{< param "DEFAULT_ENGINE" >}} isn't running, the UI and metrics aren't available at `0.0.0.0:12345/metrics`.
+
+### Run the {{% param "PRODUCT_NAME" %}} Engine extension
+
+You can also run the {{< param "OTEL_ENGINE" >}} with the {{< param "DEFAULT_ENGINE" >}}.
+Modify your YAML configuration to include the `alloyengine` extension, which accepts a path to the {{< param "DEFAULT_ENGINE" >}} configuration and starts a {{< param "DEFAULT_ENGINE" >}} pipeline alongside the {{< param "OTEL_ENGINE" >}} pipeline.
+
+The following example shows the configuration:
+
+```yaml
+extensions:
+  basicauth/grafana_cloud:
+    client_auth:
+      username: <INSTANCE_ID>
+      password: <API_TOKEN>
+  alloyengine:
+    config:
+      file: <ALLOY_CONFIG_PATH>
+    flags:
+      server.http.listen-addr: 0.0.0.0:12345
+
+receivers:
+  otlp:
+    protocols:
+      grpc: {}
+      http: {}
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 512
+
+exporters:
+  otlphttp/grafana_cloud:
+    endpoint: <GRAFANA_URL>
+    auth:
+      authenticator: basicauth/grafana_cloud
+
+service:
+  extensions: [basicauth/grafana_cloud, alloyengine]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlphttp/grafana_cloud]
+```
+
+Replace the following:
+
+- _`<ALLOY_CONFIG_PATH>`_: The path to your {{< param "DEFAULT_ENGINE" >}} configuration file.
+- _`<INSTANCE_ID>`_: Your Grafana Cloud instance ID.
+- _`<API_TOKEN>`_: Your Grafana Cloud API token.
+- _`<GRAFANA_URL>`_: Your Grafana Cloud OTLP endpoint URL.
+
+This example adds the `alloyengine` block in the extension declarations and enables the extension in the `service` block.
+You can then run {{< param "PRODUCT_NAME" >}} with the exact same command as before:
+
+```shell
+alloy otel --config=<CONFIG_FILE> [<FLAGS> ...]
+```
+
+This starts both the {{< param "DEFAULT_ENGINE" >}} and {{< param "OTEL_ENGINE" >}}.
+The output of both engines is visible in the logs.
+You can access the {{< param "DEFAULT_ENGINE" >}} UI and metrics on port `12345`.
+
+## Run with {{% param "PRODUCT_NAME" %}} Helm chart
+
+TODO
+
+## Run with service installation
+
+Service installation support for systemd, launchd, and similar systems isn't included in the initial experimental release.
+Service installers will work seamlessly with the {{< param "OTEL_ENGINE" >}} as the feature matures.
+In the meantime, use the CLI or Helm options for testing.
+
+## Considerations
+
+1. **Storage configuration**: The {{< param "DEFAULT_ENGINE" >}} accepts the `--storage.path` flag to set a base directory for components to store data on disk.
+   The {{< param "OTEL_ENGINE" >}} uses the `filestorage` extension instead of a CLI flag.
+   Refer to the [upstream documentation](https://opentelemetry.io/docs/collector/resiliency/#persistent-storage-write-ahead-log---wal) for more information.
+1. **Server ports**: The {{< param "DEFAULT_ENGINE" >}} exposes its HTTP server on port `12345`.
+   The {{< param "OTEL_ENGINE" >}} exposes its HTTP server on port `8888`.
+   The {{< param "OTEL_ENGINE" >}} HTTP server doesn't expose a UI, support bundles, or reload endpoint functionality like the {{< param "DEFAULT_ENGINE" >}}.
+1. **Fleet management**: [Grafana Fleet Management](https://grafana.com/blog/opentelemetry-and-grafana-labs-whats-new-and-whats-next-in-2026/#fleet-management) isn't supported for the {{< param "OTEL_ENGINE" >}}.
+   You must define and manage the input configuration manually. 
+   Support is a planned feature for the {{< param "OTEL_ENGINE" >}}, with OpAmp already being built into our Fleet Management solution in Grafana Cloud. You can read more about this [here](https://grafana.com/blog/opentelemetry-and-grafana-labs-whats-new-and-whats-next-in-2026/#fleet-management)