Skip to content

Enhance documentation for k8sattributes processor#5341

Open
srclosson wants to merge 1 commit intomainfrom
srclosson-patch-1
Open

Enhance documentation for k8sattributes processor#5341
srclosson wants to merge 1 commit intomainfrom
srclosson-patch-1

Conversation

@srclosson
Copy link

Added details about multi-tier deployments and workarounds for k8sattributes processor.

Brief description of Pull Request

Pull Request Details

Issue(s) fixed by this Pull Request

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated

Added details about multi-tier deployments and workarounds for k8sattributes processor.
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link
Contributor

💻 Deploy preview available (Enhance documentation for k8sattributes processor):

@clayton-cornell clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Jan 27, 2026
Copy link
Contributor

@clayton-cornell clayton-cornell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some rewrite suggestions - some minor style fixes, and then some rewrites to use active voice where possible and write for positive action vs negative (example in the note... switched it around)

A {{< param "PRODUCT_NAME" >}} running as a Deployment can't detect the IP addresses of pods generating telemetry data without any of the well-known IP attributes.
If the Deployment {{< param "PRODUCT_NAME" >}} receives telemetry from {{< param "PRODUCT_NAME" >}}s deployed as DaemonSet, then some of those attributes might be missing.
As a workaround, you can configure the DaemonSet {{< param "PRODUCT_NAME" >}}s with `passthrough` set to `true`.
**Multi-tier deployments:** When using a two-tier architecture where DaemonSet {{< param "PRODUCT_NAME" >}} instances forward telemetry to a centralized Deployment {{< param "PRODUCT_NAME" >}}, the k8sattributes processor in the Deployment cannot determine which pod originally sent the telemetry. This happens because:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Multi-tier deployments:** When using a two-tier architecture where DaemonSet {{< param "PRODUCT_NAME" >}} instances forward telemetry to a centralized Deployment {{< param "PRODUCT_NAME" >}}, the k8sattributes processor in the Deployment cannot determine which pod originally sent the telemetry. This happens because:
**Multi-tier deployments:** If you use a two-tier architecture where {{< param "PRODUCT_NAME" >}} instances deployed as a DaemonSet forward telemetry to a centralized {{< param "PRODUCT_NAME" >}} Deployment, the k8sattributes processor in the Deployment can't determine which Pod originally sent the telemetry. This happens for the following reasons:

Comment on lines +66 to +68
1. The k8sattributes processor identifies pods using the connection source IP, `k8s.pod.ip`, or `k8s.pod.uid` attributes
2. When telemetry is forwarded through a DaemonSet, the Deployment sees the DaemonSet pod's IP as the source, not the original application pod's IP
3. If the telemetry doesn't already contain `k8s.pod.ip` or `k8s.pod.uid` attributes, the Deployment cannot look up the pod's Kubernetes metadata
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. The k8sattributes processor identifies pods using the connection source IP, `k8s.pod.ip`, or `k8s.pod.uid` attributes
2. When telemetry is forwarded through a DaemonSet, the Deployment sees the DaemonSet pod's IP as the source, not the original application pod's IP
3. If the telemetry doesn't already contain `k8s.pod.ip` or `k8s.pod.uid` attributes, the Deployment cannot look up the pod's Kubernetes metadata
1. The k8sattributes processor relies on the connection source IP, `k8s.pod.ip`, or `k8s.pod.uid` attributes to identify Pods.
1. When telemetry passes through a DaemonSet, the Deployment sees the DaemonSet Pod's IP instead of the original application Pod's IP.
1. The Deployment can't retrieve metadata for the original Pod unless the telemetry contains `k8s.pod.ip` or `k8s.pod.uid` attributes.

2. When telemetry is forwarded through a DaemonSet, the Deployment sees the DaemonSet pod's IP as the source, not the original application pod's IP
3. If the telemetry doesn't already contain `k8s.pod.ip` or `k8s.pod.uid` attributes, the Deployment cannot look up the pod's Kubernetes metadata

**Workaround:** Configure the DaemonSet {{< param "PRODUCT_NAME" >}} with `passthrough = true`. This ensures the DaemonSet adds Kubernetes attributes (like `k8s.namespace.name`) before forwarding, so the Deployment receives telemetry with metadata already attached. The Deployment's k8sattributes processor will then pass through the existing attributes without modification.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Workaround:** Configure the DaemonSet {{< param "PRODUCT_NAME" >}} with `passthrough = true`. This ensures the DaemonSet adds Kubernetes attributes (like `k8s.namespace.name`) before forwarding, so the Deployment receives telemetry with metadata already attached. The Deployment's k8sattributes processor will then pass through the existing attributes without modification.
To address this, configure {{< param "PRODUCT_NAME" >}} that runs as a DaemonSet with `passthrough = true`. This ensures the DaemonSet adds Kubernetes attributes, like `k8s.namespace.name`, before forwarding, so the Deployment receives telemetry with metadata already attached. The Deployment's k8sattributes processor then passes through the attributes without modification.


**Workaround:** Configure the DaemonSet {{< param "PRODUCT_NAME" >}} with `passthrough = true`. This ensures the DaemonSet adds Kubernetes attributes (like `k8s.namespace.name`) before forwarding, so the Deployment receives telemetry with metadata already attached. The Deployment's k8sattributes processor will then pass through the existing attributes without modification.

**Note:** If your {{< param "PRODUCT_NAME" >}} receives telemetry directly from application pods (not forwarded from another {{< param "PRODUCT_NAME" >}}), keep `passthrough = false` (the default).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Note:** If your {{< param "PRODUCT_NAME" >}} receives telemetry directly from application pods (not forwarded from another {{< param "PRODUCT_NAME" >}}), keep `passthrough = false` (the default).
{{< admonition type="note" >}}
You only need to set `passthrough = true` in multi-tier deployments where another {{< param "PRODUCT_NAME" >}} instance forwards the telemetry.
{{< /admonition >}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/docs Docs Squad label across all Grafana Labs repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants