Skip to content

Conversation

@Nakshatra480
Copy link

Closes #1313

Summary

Implements OTLP tracing for dora-daemon and dora-coordinator, enabling distributed tracing via Jaeger when the DORA_OTLP_ENDPOINT environment variable is set.

Changes

  • telemetry.rs: Added service name to OTLP Resource for proper service identification in Jaeger
  • lib.rs: Registered global tracer provider to enable span export
  • daemon.rs: Runtime-safe OTLP initialization with OtelGuard lifetime management
  • coordinator.rs: Runtime-safe OTLP initialization with OtelGuard lifetime management

Key Technical Details

The implementation addresses a critical async/sync challenge: OTLP uses async gRPC (Tonic) but daemon/coordinator initialize tracing before creating the tokio runtime. The solution uses conditional two-phase initialization:

  1. Check for OTLP environment variables (sync)
  2. Create tokio runtime
  3. Initialize OTLP tracing inside runtime context (async)
    Additionally, the OtelGuard must be stored for the lifetime of the runtime to prevent the tracer provider from shutting down immediately after initialization.

Testing

Tested with Jaeger all-in-one:

# Start Jaeger
docker run -d --name jaeger \
  -p 4317:4317 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

# Run dora with tracing
export DORA_OTLP_ENDPOINT="http://localhost:4317"
./target/release/dora up
./target/release/dora start examples/rust-dataflow/dataflow.yml

# View traces at http://localhost:16686

@Nakshatra480 Nakshatra480 force-pushed the feat/otlp-tracing-daemon-coordinator branch 2 times, most recently from d6e010f to 5334c76 Compare January 22, 2026 12:42
@Nakshatra480 Nakshatra480 force-pushed the feat/otlp-tracing-daemon-coordinator branch from 5334c76 to 6f5269b Compare January 22, 2026 17:24
@Nakshatra480 Nakshatra480 reopened this Jan 22, 2026
@Nakshatra480 Nakshatra480 force-pushed the feat/otlp-tracing-daemon-coordinator branch 7 times, most recently from 11ee196 to 5876a13 Compare January 23, 2026 05:31
Enables Jaeger tracing when DORA_OTLP_ENDPOINT is set. Key changes:
- Add service name to OTLP resource configuration
- Register global tracer provider for span export
- Store OtelGuard to prevent premature shutdown
- Initialize OTLP within tokio runtime context
- Move TracingBuilder import to top-level with proper feature gate

Fixes dora-rs#1313
@Nakshatra480 Nakshatra480 force-pushed the feat/otlp-tracing-daemon-coordinator branch from 5876a13 to bc7dc5f Compare January 23, 2026 05:32
@Nakshatra480
Copy link
Author

Nakshatra480 commented Jan 24, 2026

@haixuanTao @phil-opp changes are ready, pls review it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracing via DORA_JAEGER_TRACING does not work

1 participant