feat(opentelemetry) add failover (#591)

* feat(opentelemetry) add failover

* fix indentation

* update secrets

* update docs, test-values

* change endpoint type to string

* update docs

Author: timojohlo

opentelemetry/README.md

@@ -10,7 +10,7 @@ The main terminologies used in this document can be found in [core-concepts](htt
 
 OpenTelemetry is an observability framework and toolkit for creating and managing telemetry data such as metrics, logs and traces. Unlike other observability tools, OpenTelemetry is vendor and tool agnostic, meaning it can be used with a variety of observability backends, including open source tools such as _OpenSearch_ and _Prometheus_. 
 
-The focus of the plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.
+The focus of the Plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.
 
 Components included in this Plugin:
 
@@ -21,6 +21,7 @@ Components included in this Plugin:
     - [k8sevents Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)
     - [journald Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver)
     - [prometheus/internal](https://opentelemetry.io/docs/collector/internal-telemetry/)
+- [Connector](https://opentelemetry.io/docs/collector/building/connector/) 
 - [OpenSearch Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/opensearchexporter)
 
 ## Architecture
@@ -45,7 +46,7 @@ This guide provides a quick and straightforward way to use **OpenTelemetry** as
 **Step 1:**
 
 You can install the `OpenTelemetry` package in your cluster by installing it with [Helm](https://helm.sh/docs/helm/helm_install) manually or let the Greenhouse platform lifecycle do it for you automatically. For the latter, you can either:
-  1. Go to Greenhouse dashboard and select the **OpenTelemetry** plugin from the catalog. Specify the cluster and required option values.
+  1. Go to Greenhouse dashboard and select the **OpenTelemetry** Plugin from the catalog. Specify the cluster and required option values.
   2. Create and specify a `Plugin` resource in your Greenhouse central cluster according to the [examples](#examples).
 
 **Step 2:**
@@ -70,6 +71,10 @@ Based on the backend selection the telemetry data will be exporter to the backen
 
 Greenhouse regularly performs integration tests that are bundled with **OpenTelemetry**. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard. 
 
+## Failover Connector
+
+The **OpenTelemetry** Plugin comes with a [Failover Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/failoverconnector) for OpenSearch for two users. The connector will periodically try to establish a stable connection for the prefered user (`failover_username_a`) and in case of a failed try, the connector will try to establish a connection with the fallback user (`failover_username_b`). This feature can be used to secure the shipping of logs in case of expiring credentials or password rotation.
+
 ## Configuration
 
 | Name         | Description          | Type           | required           |
@@ -78,9 +83,12 @@ Greenhouse regularly performs integration tests that are bundled with **OpenTele
 `openTelemetry.logsCollector.kvmConfig.enabled`    | Activates the configuration for KVM logs (requires logsCollector to be enabled) | bool | `false` |
 `openTelemetry.logsCollector.cephConfig.enabled`    | Activates the configuration for Ceph logs (requires logsCollector to be enabled) | bool | `false` |
 `openTelemetry.metricsCollector.enabled` | Activates the standard configuration for metrics | bool | `false` |
-`openTelemetry.openSearchLogs.username` | Username for OpenSearch endpoint | secret | `false` |
-`openTelemetry.openSearchLogs.password` | Password for OpenSearch endpoint | secret | `false` | 
-`openTelemetry.openSearchLogs.endpoint` | Endpoint URL for OpenSearch      | secret | `false` | 
+`openTelemetry.openSearchLogs.failover_username_a` | Username for OpenSearch endpoint | secret | `true` |
+`openTelemetry.openSearchLogs.failover_password_a` | Password for OpenSearch endpoint | secret | `true` | 
+`openTelemetry.openSearchLogs.failover_username_b` | Second Username (as a failover) for OpenSearch endpoint | secret | `true` |
+`openTelemetry.openSearchLogs.failover_password_b` | Second Password (as a failover) for OpenSearch endpoint | secret | `true` |
+`openTelemetry.openSearchLogs.endpoint` | Endpoint URL for OpenSearch      | string | `true` | 
+`openTelemetry.openSearchLogs.index` | Name for OpenSearch index      | string | `false` | 
 `openTelemetry.region`                   | Region label for logging         | string | `false` |
 `openTelemetry.cluster`                  | Cluster label for logging        | string | `false` |
 `openTelemetry.prometheus.additionalLabels`               | Label selector for Prometheus resources to be picked-up by the operator | map | `false` | 

opentelemetry/chart/Chart.yaml

@@ -4,7 +4,7 @@
 apiVersion: v2
 appVersion: v0.116.0
 name: opentelemetry-operator
-version: 0.7.2
+version: 0.7.3
 description: OpenTelemetry Operator Helm chart for Kubernetes
 icon: https://raw.githubusercontent.com/cncf/artwork/a718fa97fffec1b9fd14147682e9e3ac0c8817cb/projects/opentelemetry/icon/color/opentelemetry-icon-color.png
 type: application

opentelemetry/chart/ci/test-values.yaml

@@ -33,8 +33,11 @@ opentelemetry-operator:
 openTelemetry:
   openSearchLogs:
     endpoint: test
-    username: test
-    password: test
+    failover_username_a: test
+    failover_password_a: test
+    failover_username_b: test
+    failover_password_b: test
+    index: test
   cluster: test
   region: test
   logsCollector:

opentelemetry/chart/templates/logs-collector.yaml

@@ -35,6 +35,8 @@ spec:
       value: "{{ .Values.openTelemetry.cluster }}"
     - name: region
       value: "{{ .Values.openTelemetry.region }}"
+    - name: index
+      value: "{{ .Values.openTelemetry.openSearchLogs.index }}"
   envFrom:
     - secretRef:
          name: otel-basic-auth
@@ -149,6 +151,17 @@ spec:
             key: log.type
             value: "k8sevents"
 
+      attributes/failover_username_a:
+        actions:
+          - action: insert
+            key: failover_username_opensearch
+            value: ${failover_username_a}
+      attributes/failover_username_b:
+        actions:
+          - action: insert
+            key: failover_username_opensearch
+            value: ${failover_username_b}
+
       transform/journal:
         error_mode: ignore
         log_statements:
@@ -288,28 +301,45 @@ spec:
     exporters:
       debug:
         verbosity: basic
-      opensearch/logs:
+      opensearch/failover_a:
         http:
           auth:
-            authenticator: basicauth
+            authenticator: basicauth/failover_a
           endpoint: {{ .Values.openTelemetry.openSearchLogs.endpoint }}
-        logs_index: ${username}-datastream
+        logs_index: ${index}-datastream
+      opensearch/failover_b:
+        http:
+          auth:
+            authenticator: basicauth/failover_b
+          endpoint: {{ .Values.openTelemetry.openSearchLogs.endpoint }}
+        logs_index: ${index}-datastream
 
       prometheus:
         endpoint: 0.0.0.0:9999
 
     extensions:
-      basicauth:
+      basicauth/failover_a:
         client_auth:
-          username: ${username}
-          password: ${password}
+          username: ${failover_username_a}
+          password: ${failover_password_a}
+      basicauth/failover_b:
+        client_auth:
+          username: ${failover_username_b}
+          password: ${failover_password_b}
 
     connectors:
       forward: {}
-
+      failover:
+        priority_levels:
+          - [logs/failover_a]
+          - [logs/failover_b]
+        retry_interval: 2m
+        retry_gap: 30s
+        max_retries: 0
     service:
       extensions:
-        - basicauth
+        - basicauth/failover_a
+        - basicauth/failover_b
 {{- if .Values.openTelemetry.prometheus.podMonitor.enabled }}
       telemetry:
         metrics:
@@ -320,7 +350,15 @@ spec:
         logs/forward:
           receivers: [forward]
           processors: [batch]
-          exporters: [opensearch/logs]
+          exporters: [failover]
+        logs/failover_a:
+          receivers: [failover]
+          processors: [attributes/failover_username_a]
+          exporters: [opensearch/failover_a]
+        logs/failover_b:
+          receivers: [failover]
+          processors: [attributes/failover_username_b]
+          exporters: [opensearch/failover_b]
         logs/containerd:
           receivers: [filelog/containerd]
           processors: [k8sattributes,attributes/cluster,transform/ingress]

opentelemetry/chart/templates/secret.yaml

@@ -13,5 +13,8 @@ metadata:
   {{ toYaml .Values.openTelemetry.customLabels | nindent 4 }}
   {{- end }}
 data:
-   password: {{ .Values.openTelemetry.openSearchLogs.password | b64enc | quote }}
-   username: {{ .Values.openTelemetry.openSearchLogs.username | b64enc | quote }}
+   failover_username_a: {{ .Values.openTelemetry.openSearchLogs.failover_username_a | b64enc | quote }}
+   failover_password_a: {{ .Values.openTelemetry.openSearchLogs.failover_password_a | b64enc | quote }}
+   failover_username_b: {{ .Values.openTelemetry.openSearchLogs.failover_username_b | b64enc | quote }}
+   failover_password_b: {{ .Values.openTelemetry.openSearchLogs.failover_password_b | b64enc | quote }}
+   

opentelemetry/chart/values.yaml

@@ -41,8 +41,11 @@ openTelemetry:
 
   openSearchLogs:
     endpoint:
-    username:
-    password:
+    failover_username_a:
+    failover_password_a:
+    failover_username_b:
+    failover_password_b:
+    index:
   cluster:
   region:
   logsCollector:

opentelemetry/plugindefinition.yaml

@@ -6,14 +6,14 @@ kind: PluginDefinition
 metadata:
     name: opentelemetry
 spec:
-    version: 0.7.2
+    version: 0.7.3
     displayName: OpenTelemetry
     description: Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs.
     icon: https://raw.githubusercontent.com/cloudoperators/greenhouse-extensions/main/opentelemetry/logo.png
     helmChart:
         name: opentelemetry-operator
         repository: oci://ghcr.io/cloudoperators/greenhouse-extensions/charts
-        version: 0.7.2
+        version: 0.7.3
     options:
         - default: true
           description: Activates the standard configuration for logs
@@ -36,17 +36,29 @@ spec:
           required: false
           type: bool
         - description: Username for OpenSearch endpoint
-          name: openTelemetry.openSearchLogs.username
+          name: openTelemetry.openSearchLogs.failover_username_a
           required: true
-          type: secret
+          type: string
         - description: Password for OpenSearch endpoint
-          name: openTelemetry.openSearchLogs.password
+          name: openTelemetry.openSearchLogs.failover_password_a
+          required: true
+          type: secret
+        - description: Second Username (as a failover) for OpenSearch endpoint
+          name: openTelemetry.openSearchLogs.failover_username_b
+          required: true
+          type: secret
+        - description: Second Password (as a failover) for OpenSearch endpoint
+          name: openTelemetry.openSearchLogs.failover_password_b
           required: true
           type: secret
         - description: Endpoint URL for OpenSearch
           name: openTelemetry.openSearchLogs.endpoint
           required: false
           type: string
+        - description: Name for OpenSearch index
+          name: openTelemetry.openSearchLogs.index
+          required: false
+          type: string
         - description: Region label for logging
           name: openTelemetry.region
           required: false