Router ignores incoming traceparent header in production mode but works with --dev

[gianielsevier]

Router Version: v2.10.0

Description:
The router receives traceparent headers from clients and records them as span attributes, but does not use them as the parent context for traces. This results in the router creating new traces instead of continuing existing ones, breaking distributed tracing. The issue does NOT occur when running with --dev flag.

Configuration:

telemetry:
  instrumentation:
    spans:
      mode: spec_compliant
  exporters:
    tracing:
      experimental_response_trace_id:
        enabled: true
        header_name: "apollo-trace-id"
      propagation:
        trace_context: true
        baggage: false
      common:
        parent_based_sampler: true
        sampler: always_on
        resource:
          "service.name": "prod-apollo-router"
          "environment.name": "prod"
      otlp:
        enabled: true
        endpoint: https://otlp.nr-data.net:4317
        protocol: grpc
        grpc:
          metadata:
            api-key: <redacted>

Headers configuration:

headers:
  all:
    request:
      - propagate:
          matching: .*

Observed Behavior:
Router receives traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
Router records it as span attribute: "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
Router generates NEW trace ID instead of using 4bf92f3577b34da6a3ce929d0e0e4736
Router does NOT propagate traceparent to subgraphs
Subgraphs (Spring Boot with OpenTelemetry + New Relic) receive no trace context

Expected Behavior:
Router should use incoming traceparent as parent context
Router should continue the trace with the same trace ID
Router should propagate trace context to subgraphs
apollo-trace-id response header should match incoming trace ID

Workaround:
Running with --dev flag makes trace propagation work correctly.

Environment:
Deployment: Kubernetes (EKS)
Router image: ghcr.io/apollographql/router:v2.10.0
Subgraphs: Spring Boot with OpenTelemetry and New Relic agent

[gianielsevier]

I found the real issue.
When I run the router without --dev and --log error, tracing propagation doesn't work, the router ignores incoming traceparent headers and creates new traces instead of continuing existing ones.
If I change the log level to --log info or --log debug, tracing propagation works correctly and the router properly continues traces from clients.
This means log level is somehow affecting OpenTelemetry trace context extraction, which seems like a bug since logging and tracing should be independent.

[abernix]

So it's worth noting that logging and tracing are not completely independent concepts because OpenTelemetry does actually keep track of these together (logs are attached to spans, which is really useful when you need that context in a trace viewer!), but you're right that trace propagation not working in this particular case would be unexpected and it sounds like you're onto a bug.

Was there a previous version where this was working for you before it stopped? Or does this exist in all versions you've tested. If it's a regression, your help tracking it down would be useful since you've got the reproduction up and running.