Skip to content

Mark Pods deployed by k8s collector with "sw.k8s.deployedbycollector" #1242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
etichy merged 5 commits into from
Dec 17, 2025
Merged

Mark Pods deployed by k8s collector with "sw.k8s.deployedbycollector" #1242

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e7d035b
Mark Pods deployed by k8s collector with "sw.k8s.deployedbycollector"
pstranak-sw Dec 15, 2025
6883bc8
Adjust version
etichy Dec 17, 2025
4e16ef5
Merge branch 'master' into markPodsDeployedByK8sCollector
etichy Dec 17, 2025
2a45325
Fix
etichy Dec 17, 2025
8e031f4
Merge remote-tracking branch 'origin/markPodsDeployedByK8sCollector' …
etichy Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion deploy/helm/events-collector-config.yaml
View file
Open in desktop
Copy link
Contributor Author

@pstranak-sw pstranak-sw Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in this file depend on solarwinds/solarwinds-otel-collector-contrib#220

Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ processors:
transform/entity_attributes:
log_statements:
- statements:
# do not create/update endities from events
# do not create/update entities from events
- set(resource.attributes["sw.entity.noupdate"], "true")

- set(resource.attributes["k8s.pod.name"], resource.attributes["k8s.object.name"]) where resource.attributes["k8s.object.kind"] == "Pod"
Expand Down Expand Up @@ -371,6 +371,18 @@ processors:
statements:
- set(resource.attributes["sw.k8s.service.type"], log.body["spec"]["type"]) where log.body["spec"]["type"] != nil

transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
log_statements:
- conditions:
- resource.attributes["sw.k8s.log.type"] == "entitystateevent"
and log.attributes["otel.entity.attributes"] != nil
and log.attributes["otel.entity.attributes"]["sw.k8s.deployedbycollector"] == true
and log.attributes["otel.entity.id"] != nil
and log.attributes["otel.entity.id"]["k8s.namespace.name"] != "{{ .Release.Namespace }}"
statements:
- delete_key(log.attributes["otel.entity.attributes"], "sw.k8s.deployedbycollector")

transform/set_labels_and_annotations_for_entities:
error_mode: ignore
log_statements:
Expand Down Expand Up @@ -746,6 +758,7 @@ service:
- resource/manifest
- resourcedetection/providers
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- groupbyattrs/serviceendpointsmapping
- transform/serviceendpointsmapping-renamepodip
- k8sattributes
Expand Down Expand Up @@ -793,6 +806,7 @@ service:
- resource/manifest
- k8sattributes
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
{{- if eq (include "isNamespacesFilterEnabled" .) "true" }}
- filter/namespaces
{{- end }}
Expand Down
39 changes: 38 additions & 1 deletion deploy/helm/metrics-collector-config.yaml
View file
Open in desktop
Copy link
Contributor Author

@pstranak-sw pstranak-sw Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to test whether this actually works, because the all container metrics (kube_pod_container_*) collected by this collector are marked with sw.entity.noupdate. I think it doesn't actually do anything. But if it does, it would make this change non-functional. But it would also mean that collecting container statuses, images, "init" info, etc. from metrics would not work either.

Copy link
Contributor

@gantrior gantrior Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why do we instrument telemetry by anything at all here? We updated all pods that we deploy and all pods that are deployed by subcharts, so why do we need to update k8sattributes config at all?

Copy link
Contributor Author

@pstranak-sw pstranak-sw Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to extract a resource attribute out of the label. There are multiple ways how to achieve that. I used this one as it makes it unified with what I did in the manifest collection pipeline. And it allowed to me to add also the sanity check for to sure the resource attribute is not set in case of incorrect namespace.

Copy link
Contributor

@gantrior gantrior Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, clarified async. We will keep this backup there until we make sure that event collector and ingenstion of entity state events from manifest cannot be disabled.

Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,34 @@ extensions:

processors:
k8sattributes:
{{ include "common.k8s-instrumentation" . | indent 4 }}
auth_type: "serviceAccount"
Copy link
Contributor

@etichy etichy Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this copy paste, but probably there's no other way.

Copy link
Contributor Author

@pstranak-sw pstranak-sw Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd actually prefer to remove most of the deduplicated code, as it only complicates investigations of what each pipeline does and what data it really depends on. Not all of the shared configuration is really used by all of the pipelines, that reference it.

passthrough: false
extract:
metadata:
- k8s.pod.name
- k8s.deployment.name
- k8s.replicaset.name
- k8s.daemonset.name
- k8s.job.name
- k8s.cronjob.name
- k8s.statefulset.name
- k8s.node.name
labels:
- tag_name: sw.k8s.deployedbycollector
key: swo.cloud.solarwinds.com/deployed-with-k8s-collector
from: pod
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.name
- from: resource_attribute
name: k8s.namespace.name
- sources:
- from: resource_attribute
name: k8s.pod.ip
- from: resource_attribute
name: k8s.namespace.name

memory_limiter:
{{ toYaml .Values.otel.metrics.memory_limiter | indent 4 }}
transform:
Expand Down Expand Up @@ -939,6 +966,15 @@ processors:
)'
{{- end }}

transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
metric_statements:
- conditions:
- resource.attributes["sw.k8s.deployedbycollector"] == "true"
and resource.attributes["k8s.namespace.name"] != "{{ .Release.Namespace }}"
statements:
- delete_key(resource.attributes, "sw.k8s.deployedbycollector")

connectors:
forward/prometheus: {}
forward/metric-exporter: {}
Expand Down Expand Up @@ -1241,6 +1277,7 @@ service:
- groupbyattrs/all
- resource/metrics
- k8sattributes
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- transform/cleanup_attributes_for_nonexisting_entities
{{- if eq (include "isNamespacesFilterEnabled" .) "true" }}
- filter/namespaces
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ app.kubernetes.io/managed-by: {{ .Release.Name }}
helm.sh/chart: {{ include "common.chart" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
swo.cloud.solarwinds.com/deployed-with-k8s-collector: "true"
{{- if .Values.commonLabels}}
{{ toYaml .Values.commonLabels }}
{{- end }}
Expand Down
47 changes: 47 additions & 0 deletions deploy/helm/tests/__snapshot__/events-collector-config-map_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,16 @@ Custom events filter with new syntax:
log_statements:
- statements:
- set(log.attributes["sw.namespace"], "sw.events.inframon.k8s")
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
log_statements:
- conditions:
- resource.attributes["sw.k8s.log.type"] == "entitystateevent" and log.attributes["otel.entity.attributes"]
!= nil and log.attributes["otel.entity.attributes"]["sw.k8s.deployedbycollector"]
== true and log.attributes["otel.entity.id"] != nil and log.attributes["otel.entity.id"]["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(log.attributes["otel.entity.attributes"], "sw.k8s.deployedbycollector")
transform/scope:
log_statements:
- statements:
Expand Down Expand Up @@ -948,6 +958,7 @@ Custom events filter with new syntax:
- resource/manifest
- resourcedetection/providers
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- groupbyattrs/serviceendpointsmapping
- transform/serviceendpointsmapping-renamepodip
- k8sattributes
Expand All @@ -974,6 +985,7 @@ Custom events filter with new syntax:
- resource/manifest
- k8sattributes
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
receivers:
- swok8sobjects/keepalive
logs/serviceendpointsmapping:
Expand Down Expand Up @@ -1541,6 +1553,16 @@ Custom events filter with old syntax:
log_statements:
- statements:
- set(log.attributes["sw.namespace"], "sw.events.inframon.k8s")
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
log_statements:
- conditions:
- resource.attributes["sw.k8s.log.type"] == "entitystateevent" and log.attributes["otel.entity.attributes"]
!= nil and log.attributes["otel.entity.attributes"]["sw.k8s.deployedbycollector"]
== true and log.attributes["otel.entity.id"] != nil and log.attributes["otel.entity.id"]["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(log.attributes["otel.entity.attributes"], "sw.k8s.deployedbycollector")
transform/scope:
log_statements:
- statements:
Expand Down Expand Up @@ -1962,6 +1984,7 @@ Custom events filter with old syntax:
- resource/manifest
- resourcedetection/providers
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- groupbyattrs/serviceendpointsmapping
- transform/serviceendpointsmapping-renamepodip
- k8sattributes
Expand All @@ -1988,6 +2011,7 @@ Custom events filter with old syntax:
- resource/manifest
- k8sattributes
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
receivers:
- swok8sobjects/keepalive
logs/serviceendpointsmapping:
Expand Down Expand Up @@ -2548,6 +2572,16 @@ Events config should match snapshot when using default values:
log_statements:
- statements:
- set(log.attributes["sw.namespace"], "sw.events.inframon.k8s")
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
log_statements:
- conditions:
- resource.attributes["sw.k8s.log.type"] == "entitystateevent" and log.attributes["otel.entity.attributes"]
!= nil and log.attributes["otel.entity.attributes"]["sw.k8s.deployedbycollector"]
== true and log.attributes["otel.entity.id"] != nil and log.attributes["otel.entity.id"]["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(log.attributes["otel.entity.attributes"], "sw.k8s.deployedbycollector")
transform/scope:
log_statements:
- statements:
Expand Down Expand Up @@ -2968,6 +3002,7 @@ Events config should match snapshot when using default values:
- resource/manifest
- resourcedetection/providers
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- groupbyattrs/serviceendpointsmapping
- transform/serviceendpointsmapping-renamepodip
- k8sattributes
Expand All @@ -2994,6 +3029,7 @@ Events config should match snapshot when using default values:
- resource/manifest
- k8sattributes
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
receivers:
- swok8sobjects/keepalive
logs/serviceendpointsmapping:
Expand Down Expand Up @@ -3422,6 +3458,16 @@ Events config should not contain manifest collection pipeline when disabled:
log_statements:
- statements:
- set(log.attributes["sw.namespace"], "sw.events.inframon.k8s")
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
log_statements:
- conditions:
- resource.attributes["sw.k8s.log.type"] == "entitystateevent" and log.attributes["otel.entity.attributes"]
!= nil and log.attributes["otel.entity.attributes"]["sw.k8s.deployedbycollector"]
== true and log.attributes["otel.entity.id"] != nil and log.attributes["otel.entity.id"]["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(log.attributes["otel.entity.attributes"], "sw.k8s.deployedbycollector")
transform/scope:
log_statements:
- statements:
Expand Down Expand Up @@ -3677,6 +3723,7 @@ Events config should not contain manifest collection pipeline when disabled:
- resource/manifest
- resourcedetection/providers
- k8seventgeneration
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- groupbyattrs/serviceendpointsmapping
- transform/serviceendpointsmapping-renamepodip
- k8sattributes
Expand Down
13 changes: 13 additions & 0 deletions deploy/helm/tests/__snapshot__/metrics-collector-config-map-fargate_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,10 @@ Metrics config should match snapshot when using default values:
k8sattributes:
auth_type: serviceAccount
extract:
labels:
- from: pod
key: swo.cloud.solarwinds.com/deployed-with-k8s-collector
tag_name: sw.k8s.deployedbycollector
metadata:
- k8s.pod.name
- k8s.deployment.name
Expand Down Expand Up @@ -1497,6 +1501,14 @@ Metrics config should match snapshot when using default values:
- statements:
- set(resource.attributes["sw.entity.noupdate"], "true") where resource.attributes["k8s.node.name"]
!= nil
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
metric_statements:
- conditions:
- resource.attributes["sw.k8s.deployedbycollector"] == "true" and resource.attributes["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(resource.attributes, "sw.k8s.deployedbycollector")
transform/scope:
metric_statements:
- statements:
Expand Down Expand Up @@ -1763,6 +1775,7 @@ Metrics config should match snapshot when using default values:
- groupbyattrs/all
- resource/metrics
- k8sattributes
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- transform/cleanup_attributes_for_nonexisting_entities
- filter/remove_temporary_metrics
receivers:
Expand Down
39 changes: 39 additions & 0 deletions deploy/helm/tests/__snapshot__/metrics-collector-config-map_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,10 @@ Metrics config should match snapshot when fargate is enabled:
k8sattributes:
auth_type: serviceAccount
extract:
labels:
- from: pod
key: swo.cloud.solarwinds.com/deployed-with-k8s-collector
tag_name: sw.k8s.deployedbycollector
metadata:
- k8s.pod.name
- k8s.deployment.name
Expand Down Expand Up @@ -1497,6 +1501,14 @@ Metrics config should match snapshot when fargate is enabled:
- statements:
- set(resource.attributes["sw.entity.noupdate"], "true") where resource.attributes["k8s.node.name"]
!= nil
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
metric_statements:
- conditions:
- resource.attributes["sw.k8s.deployedbycollector"] == "true" and resource.attributes["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(resource.attributes, "sw.k8s.deployedbycollector")
transform/scope:
metric_statements:
- statements:
Expand Down Expand Up @@ -1763,6 +1775,7 @@ Metrics config should match snapshot when fargate is enabled:
- groupbyattrs/all
- resource/metrics
- k8sattributes
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- transform/cleanup_attributes_for_nonexisting_entities
- filter/remove_temporary_metrics
receivers:
Expand Down Expand Up @@ -2257,6 +2270,10 @@ Metrics config should match snapshot when using Prometheus url with extra_scrape
k8sattributes:
auth_type: serviceAccount
extract:
labels:
- from: pod
key: swo.cloud.solarwinds.com/deployed-with-k8s-collector
tag_name: sw.k8s.deployedbycollector
metadata:
- k8s.pod.name
- k8s.deployment.name
Expand Down Expand Up @@ -3302,6 +3319,14 @@ Metrics config should match snapshot when using Prometheus url with extra_scrape
- statements:
- set(resource.attributes["sw.entity.noupdate"], "true") where resource.attributes["k8s.node.name"]
!= nil
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
metric_statements:
- conditions:
- resource.attributes["sw.k8s.deployedbycollector"] == "true" and resource.attributes["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(resource.attributes, "sw.k8s.deployedbycollector")
transform/scope:
metric_statements:
- statements:
Expand Down Expand Up @@ -3503,6 +3528,7 @@ Metrics config should match snapshot when using Prometheus url with extra_scrape
- groupbyattrs/all
- resource/metrics
- k8sattributes
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- transform/cleanup_attributes_for_nonexisting_entities
- filter/remove_temporary_metrics
receivers:
Expand Down Expand Up @@ -4004,6 +4030,10 @@ Metrics config should match snapshot when using default values:
k8sattributes:
auth_type: serviceAccount
extract:
labels:
- from: pod
key: swo.cloud.solarwinds.com/deployed-with-k8s-collector
tag_name: sw.k8s.deployedbycollector
metadata:
- k8s.pod.name
- k8s.deployment.name
Expand Down Expand Up @@ -5049,6 +5079,14 @@ Metrics config should match snapshot when using default values:
- statements:
- set(resource.attributes["sw.entity.noupdate"], "true") where resource.attributes["k8s.node.name"]
!= nil
transform/remove_deployed_by_collector_label_if_from_different_namespace:
error_mode: ignore
metric_statements:
- conditions:
- resource.attributes["sw.k8s.deployedbycollector"] == "true" and resource.attributes["k8s.namespace.name"]
!= "NAMESPACE"
statements:
- delete_key(resource.attributes, "sw.k8s.deployedbycollector")
transform/scope:
metric_statements:
- statements:
Expand Down Expand Up @@ -5235,6 +5273,7 @@ Metrics config should match snapshot when using default values:
- groupbyattrs/all
- resource/metrics
- k8sattributes
- transform/remove_deployed_by_collector_label_if_from_different_namespace
- transform/cleanup_attributes_for_nonexisting_entities
- filter/remove_temporary_metrics
receivers:
Expand Down
3 changes: 2 additions & 1 deletion deploy/helm/tests/__snapshot__/target-allocator_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ Target Allocator deployment spec should match snapshot:
template:
metadata:
annotations:
checksum/config: 3c56cadd046598061587a72aef09bfea163380638912acf86c47a1a39c7bce6d
checksum/config: 9c6d7cded0e280222d07c522b1d2036c7671ffac5ff89ba8abdaedfc7ff8393e
labels:
app: swi-k8s-opentelemet-swo-k8s-collector-prometheus-discovery-ta
app.kubernetes.io/instance: swi-k8s-opentelemetry-collector-swo-k8s-collector
app.kubernetes.io/managed-by: swi-k8s-opentelemetry-collector
app.kubernetes.io/part-of: swo-k8s-collector
app.kubernetes.io/version: 1.0.0
helm.sh/chart: swo-k8s-collector-1.0.0
swo.cloud.solarwinds.com/deployed-with-k8s-collector: "true"
spec:
affinity:
nodeAffinity:
Expand Down
Loading
Loading