docs: Iterate on the OTel Engine Documentation

docs/sources/_index.md

@@ -11,6 +11,9 @@ cascade:
   BEYLA_VERSION: v2.8.5
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
+  FULL_OTEL_ENGINE: Alloy OpenTelemetry Engine
+  OTEL_ENGINE: OTel Engine
+  DEFAULT_ENGINE: Default Engine
 hero:
   title: Grafana Alloy
   level: 1

docs/sources/_index.md.t

@@ -11,6 +11,9 @@ cascade:
   BEYLA_VERSION: v2.8.5
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
+  FULL_OTEL_ENGINE: Alloy OpenTelemetry Engine
+  OTEL_ENGINE: OTel Engine
+  DEFAULT_ENGINE: Default Engine
 hero:
   title: Grafana Alloy
   level: 1

docs/sources/open-telemetry/_index.md

@@ -0,0 +1,146 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/opentelemetry/
+description: Grafana Alloy is a flexible, high performance, vendor-neutral distribution of the OTel Collector
+menuTitle: Open Telemetry
+title: Open Telemetry With Grafana Alloy
+_build:
+  list: false
+noindex: true
+weight: 10
+---
+
+## Overview
+
+{{< param "FULL_PRODUCT_NAME" >}} combines the Prometheus-native, production-grade collection features of {{< param "PRODUCT_NAME" >}} with the broad ecosystem and standards of OpenTelemetry.
+The {{< param "FULL_OTEL_ENGINE" >}} is a bundled OpenTelemetry Collector distribution embedded within {{< param "PRODUCT_NAME" >}} that lets you run {{< param "PRODUCT_NAME" >}} as a fully compatible OTel Collector while retaining access to existing features and integrations.
+
+{{< docs/shared lookup="stability/experimental_otel.md" source="alloy" version="<ALLOY_VERSION>" >}}
+
+## Terminology
+
+{{< param "PRODUCT_NAME" >}} supports two runtime engines and an extension:
+
+- **{{< param "DEFAULT_ENGINE" >}}**: The existing {{< param "PRODUCT_NAME" >}} runtime and configuration syntax.
+  This remains the non-breaking, primary experience for existing {{< param "PRODUCT_NAME" >}} users.
+
+- **{{< param "OTEL_ENGINE" >}}**: The new runtime that runs our OpenTelemetry distribution inside {{< param "PRODUCT_NAME" >}}, using standard [upstream collector YAML configuration](https://opentelemetry.io/docs/collector/configuration/) for pipelines and components.
+
+- **{{< param "PRODUCT_NAME" >}} Engine extension**: An OpenTelemetry Collector extension that allows you to run both the {{< param "DEFAULT_ENGINE" >}} and the {{< param "OTEL_ENGINE" >}} in parallel.
+
+## Included components
+
+The {{< param "OTEL_ENGINE" >}} bundle includes:
+
+- Standard components from the OpenTelemetry Collector core and contributor repositories
+- The `alloyengine` extension
+
+The following sections list all included components:
+
+{{< collapse title="Extensions" >}}
+
+- [alloyengine](https://github.com/grafana/alloy/tree/main/extension/alloyengine)
+- [basicauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension)
+- [bearertokenauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/bearertokenauthextension)
+- [headerssetter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextension)
+- [healthcheck](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextension)
+- [jaegerremotesampling](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/jaegerremotesampling)
+- [oauth2clientauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/oauth2clientauthextension)
+- [pprof](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/pprofextension)
+- [sigv4auth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/sigv4authextension)
+- [filestorage](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage)
+- [zpages](https://github.com/open-telemetry/opentelemetry-collector/tree/main/extension/zpagesextension)
+
+{{< /collapse >}}
+
+{{< collapse title="Configuration Providers" >}}
+
+- [env](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/envprovider)
+- [file](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/fileprovider)
+- [http](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/httpprovider)
+- [https](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/httpsprovider)
+- [yaml](https://github.com/open-telemetry/opentelemetry-collector/tree/main/confmap/provider/yamlprovider)
+
+{{< /collapse >}}
+
+{{< collapse title="Receivers" >}}
+
+- [awscloudwatch](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver)
+- [awsecscontainermetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsecscontainermetricsreceiver)
+- [awss3](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awss3receiver)
+- [cloudflare](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/cloudflarereceiver)
+- [datadog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver)
+- [faro](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/faroreceiver)
+- [filelog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver)
+- [filestats](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filestatsreceiver)
+- [fluentforward](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver)
+- [googlecloudpubsub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/googlecloudpubsubreceiver)
+- [hostmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver)
+- [influxdb](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/influxdbreceiver)
+- [jaeger](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver)
+- [kafka](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkareceiver)
+- [solace](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/solacereceiver)
+- [splunkhec](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver)
+- [syslog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/syslogreceiver)
+- [tcplog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver)
+- [vcenter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/vcenterreceiver)
+- [zipkin](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver)
+- [otlp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver)
+{{< /collapse >}}
+
+{{< collapse title="Connectors" >}}
+- [count](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector)
+- [grafanacloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/grafanacloudconnector)
+- [servicegraph](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector)
+- [spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector)
+- [forward](https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector)
+
+{{< /collapse >}}
+
+{{< collapse title="Processors" >}}
+
+- [attributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor)
+- [cumulativetodelta](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor)
+- [deltatocumulative](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor)
+- [filter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor)
+- [groupbyattrs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbyattrsprocessor)
+- [interval](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor)
+- [k8sattributes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor)
+- [metricstarttime](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstarttimeprocessor)
+- [probabilisticsampler](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor)
+- [resource](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor)
+- [resourcedetection](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor)
+- [span](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanprocessor)
+- [tailsampling](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
+- [transform](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor)
+- [batch](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor)
+- [memorylimiter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor)
+
+{{< /collapse >}}
+
+{{< collapse title="Exporters" >}}
+
+- [awss3](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awss3exporter)
+- [faro](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/faroexporter)
+- [file](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/fileexporter)
+- [googlecloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter)
+- [googlecloudpubsub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudpubsubexporter)
+- [kafka](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter)
+- [loadbalancing](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter)
+- [prometheus](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter)
+- [prometheusremotewrite](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter)
+- [splunkhec](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter)
+- [syslog](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/syslogexporter)
+- [zipkin](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/zipkinexporter)
+- [debug](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter)
+- [nop](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/nopexporter)
+- [otlp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter)
+- [otlphttp](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter)
+
+{{< /collapse >}}
+
+To view the full list of components and their versioning, refer to the [OCB manifest](https://github.com/grafana/alloy/blob/main/collector/builder-config.yaml)
+
+## Next steps
+
+- [Get started](get-started.md) with the {{< param "OTEL_ENGINE" >}}
+- [OTel CLI](../reference/cli/otel.md) reference documentation

docs/sources/open-telemetry/get-started.md

@@ -0,0 +1,134 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/opentelemetry/get-started
+description: Getting Started with the Alloy OpenTelemetry Engine 
+menuTitle: Get Started
+title: Get Started with the Alloy OpenTelemetry Engine
+_build:
+  list: false
+noindex: true
+weight: 10
+---
+
+# Get Started with the {{% param "FULL_OTEL_ENGINE" %}}
+
+You can run the {{< param "OTEL_ENGINE" >}} using the CLI, Helm chart, or service installation.
+
+## Run with the CLI
+
+The {{< param "OTEL_ENGINE" >}} is available under the {{< param "PRODUCT_NAME" >}} `otel` command.
+The CLI is the easiest way to experiment locally or on a single host.
+Refer to the [OTel CLI](../reference/cli/otel.md) documentation for more information.
+
+The following example configuration file accepts OTLP and sends it to the Grafana Cloud OTLP gateway.
+You can adapt the endpoint and credentials for your region and tenant.
+For more information about where to find your `INSTANCE_ID`, `API_TOKEN`, and `GRAFANA_URL`, refer to [Send data using OpenTelemetry Protocol](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/).
+
+```yaml
+extensions:
+  basicauth/grafana_cloud:
+    client_auth:
+      username: INSTANCE_ID
+      password: API_TOKEN
+
+receivers:
+  otlp:
+    protocols:
+      grpc: {}
+      http: {}
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 512
+
+exporters:
+  otlphttp/grafana_cloud:
+    endpoint: GRAFANA_URL
+    auth:
+      authenticator: basicauth/grafana_cloud
+
+service:
+  extensions: [basicauth/grafana_cloud]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlphttp/grafana_cloud]
+```
+
+To start the {{< param "OTEL_ENGINE" >}}, run the following command:
+
+```shell
+alloy otel --config=<CONFIG_FILE> [<FLAGS> ...] 
+```
+
+{{< param "PRODUCT_NAME" >}} then accepts incoming OTLP data on `0.0.0.0:4317` for gRPC and `0.0.0.0:4318` for HTTP requests.
+Metrics are also available on the default collector port and endpoint at `0.0.0.0:8888/metrics`.
+Since the {{< param "DEFAULT_ENGINE" >}} isn't running, the UI and metrics aren't available at `0.0.0.0:12345/metrics`.
+
+### Run the {{% param "PRODUCT_NAME" %}} Engine extension
+
+You can also run the {{< param "OTEL_ENGINE" >}} with the {{< param "DEFAULT_ENGINE" >}}.
+Modify your YAML configuration to include the `alloyengine` extension, which accepts a path to the {{< param "DEFAULT_ENGINE" >}} configuration and starts a {{< param "DEFAULT_ENGINE" >}} pipeline alongside the {{< param "OTEL_ENGINE" >}} pipeline.
+
+The following example shows the configuration:
+
+```yaml
+extensions:
+  basicauth/grafana_cloud:
+    client_auth:
+      username: INSTANCE_ID
+      password: API_TOKEN
+  alloyengine:
+    config:
+      file: path/to/alloy-config.alloy
+    flags:
+      server.http.listen-addr: 0.0.0.0:12345
+      stability.level: experimental
+
+receivers:
+  otlp:
+    protocols:
+      grpc: {}
+      http: {}
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 512
+
+exporters:
+  otlphttp/grafana_cloud:
+    endpoint: GRAFANA_URL
+    auth:
+      authenticator: basicauth/grafana_cloud
+
+service:
+  extensions: [basicauth/grafana_cloud, alloyengine]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlphttp/grafana_cloud]
+```
+
+This example adds the `alloyengine` block in the extension declarations and enables the extension in the `service` block.
+You can then run {{< param "PRODUCT_NAME" >}} with the exact same command as before:
+
+```shell
+alloy otel --config=<CONFIG_FILE> [<FLAGS> ...] 
+```
+
+This starts both the {{< param "DEFAULT_ENGINE" >}} and {{< param "OTEL_ENGINE" >}}.
+The output of both engines is visible in the logs.
+You can access the {{< param "DEFAULT_ENGINE" >}} UI and metrics on port `12345`.
+
+## Run with {{% param "PRODUCT_NAME" %}} Helm chart
+
+TODO
+
+## Run with service installation
+
+Service installation support for systemd, launchd, and similar systems isn't included in the initial experimental release.
+Service installers will work seamlessly with the {{< param "OTEL_ENGINE" >}} as the feature progresses.
+In the meantime, use the CLI or Helm options for testing.