Otel internal metrics not working

[martinrw]

What's wrong?

We are sending all of our metrics via otel, the normal metrics appear to be working but when we try to enable the internal ones we get an error like:

time=2026-02-03T11:30:05.179Z level=INFO msg="Grafana Beyla" Version=v2.7.5 Revision=cae4a4f "OpenTelemetry SDK Version"=1.38.0
time=2026-02-03T11:30:05.182Z level=ERROR msg="wrong Beyla configuration" error="you can't enable OTEL internal metrics without enabling OTEL metrics"

Steps to reproduce

Configure Beyla as described in the configmap below and restart the service

System information

No response

Software version

v2.7.5

Configuration

application_metrics:
  enabled: false
attributes:
  kubernetes:
    enable: true
discovery:
  instrument:
  - k8s_pod_labels:
      instrument: beyla
ebpf:
  context_propagation: headers
filter:
  application:
    db_operation_name:
      not_match: '{PING,CLIENT,ismaster}'
    rpc.method:
      not_match: /opentelemetry.proto.collector.metrics.v1.MetricsService/Export
  network:
    k8s_dst_owner_name:
      not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
    k8s_src_owner_name:
      not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
metrics:
  enabled: true
network:
  enabled: true
otel_metrics_export:
  enabled: true
  protocol: http/protobuf
otel_traces_export:
  endpoint: http://opentelemetry-collector.opentelemetry-collector.svc.cluster.local.:4318
  insecure_skip_verify: null
  sampler:
    arg: "0.1"
    name: traceidratio
routes:
  ignored_patterns:
  - /health
  - /status
span_metrics:
  enabled: true
internal_metrics:
  exporter: otel

Logs

time=2026-02-03T11:30:05.179Z level=INFO msg="Grafana Beyla" Version=v2.7.5 Revision=cae4a4f "OpenTelemetry SDK Version"=1.38.0
time=2026-02-03T11:30:05.182Z level=ERROR msg="wrong Beyla configuration" error="you can't enable OTEL internal metrics without enabling OTEL metrics"

[marctc]

otel_metrics_export:
enabled: true
protocol: http/protobuf

you need to define here the endpoint, otherwise this wont work (where would metrics/internal metrics be sent?)

[martinrw]

fair point, thanks for pointing it out.

I just set a value for endpoint and now the pods crash with this error

time=2026-02-03T13:43:26.424Z level=INFO msg="starting Beyla in Application Observability mode"
time=2026-02-03T13:43:26.425Z level=INFO msg="waiting for K8s metadata synchronization" component=kube.CacheSvcClient timeout=30s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x21a355c]

goroutine 28 [running]:
go.opentelemetry.io/obi/pkg/export/otel.(*InternalMetricsReporter).InformerLag(0x482c6f?, 0xc0003d7c38?)
	/src/vendor/go.opentelemetry.io/obi/pkg/export/otel/metrics_internal.go:293 +0x1c
go.opentelemetry.io/obi/pkg/components/kube.(*Store).On(0xc0003c8180, 0xc00045bcc0)
	/src/vendor/go.opentelemetry.io/obi/pkg/components/kube/store.go:228 +0x117
go.opentelemetry.io/obi/pkg/kubecache/meta.(*BaseNotifier).notifyAll(0xc000412300, 0xc00045bcc0)
	/src/vendor/go.opentelemetry.io/obi/pkg/kubecache/meta/observer.go:63 +0x174
go.opentelemetry.io/obi/pkg/kubecache/meta.(*BaseNotifier).Notify(0xc000412300, 0x305f788?)
	/src/vendor/go.opentelemetry.io/obi/pkg/kubecache/meta/observer.go:49 +0x28
go.opentelemetry.io/obi/pkg/components/kube.(*cacheSvcClient).connect(0xc000412300, {0x305f788, 0xc0002c2460})
	/src/vendor/go.opentelemetry.io/obi/pkg/components/kube/cache_svc_client.go:123 +0x24d
go.opentelemetry.io/obi/pkg/components/kube.(*cacheSvcClient).Start.func1()
	/src/vendor/go.opentelemetry.io/obi/pkg/components/kube/cache_svc_client.go:82 +0x18e
created by go.opentelemetry.io/obi/pkg/components/kube.(*cacheSvcClient).Start in goroutine 27
	/src/vendor/go.opentelemetry.io/obi/pkg/components/kube/cache_svc_client.go:64 +0x15c

If I remove:
internal_metrics:
exporter: otel
then It starts up fine... there error appears only when that but of config is defined

[marctc]

Yeah, it's know bug: open-telemetry/opentelemetry-ebpf-instrumentation#873
will be fixed in new release