Merge branch 'main' into fix/null-external-clusters

Author: itay-grudev

RELEASE.md

@@ -15,7 +15,8 @@ release (e.g. 1.17.1)
 ## Charts
 
 1. [Releasing the `cloudnative-pg` chart](#releasing-the-cloudnative-pg-chart)
-2. [Releasing `cluster` chart](#releasing-the-cluster-chart)
+2. [Releasing the `cluster` chart](#releasing-the-cluster-chart)
+3. [Releasing the `plugin-barman-cloud` chart](#releasing-the-plugin-barman-cloud)
 
 ## Releasing the `cloudnative-pg` chart
 
@@ -25,7 +26,8 @@ In order to create a new release of the `cloudnative-pg` chart, follow these ste
     ```bash
     OLD_VERSION=$(yq -r '.version' charts/cloudnative-pg/Chart.yaml)
     OLD_CNPG_VERSION=$(yq -r '.appVersion' charts/cloudnative-pg/Chart.yaml)
-    echo $OLD_VERSION
+    echo Old chart version: $OLD_VERSION
+    echo Old CNPG version: $OLD_CNPG_VERSION
     ```
 2. Decide which version to create, depending on the kind of jump of the CloudNativePG release, following semver
     semantics. For this document, let's call it `X.Y.Z`
@@ -44,8 +46,8 @@ In order to create a new release of the `cloudnative-pg` chart, follow these ste
     want to:
     1. Find the latest `cloudnative-pg` version by running:
         ```bash
-        NEW_CNPG_VERSION=$(curl  "https://api.github.com/repos/cloudnative-pg/cloudnative-pg/tags" | jq -r '.[0].name | ltrimstr("v")')
-        echo $NEW_CNPG_VERSION
+        NEW_CNPG_VERSION=$(curl -Ssl "https://api.github.com/repos/cloudnative-pg/cloudnative-pg/tags" | jq -r '.[0].name | ltrimstr("v")')
+        echo New CNPG version: $NEW_CNPG_VERSION
         ```
     2. Update `.appVersion` in the [Chart.yaml](./charts/cloudnative-pg/Chart.yaml) file
         ```bash
@@ -155,3 +157,91 @@ In order to create a new release of the `cluster` chart, follow these steps:
     helm search repo cnpg
     ```
     and be able to see the new version `X.Y.Z` as `CHART VERSION` for `cluster`
+
+## Releasing the `plugin-barman-cloud` chart
+
+In order to create a new release of the `plugin-barman-cloud` chart, follow these steps:
+
+1. Take note of the current value of the release: see `.version` in `charts/plugin-barman-cloud/Chart.yaml`
+    ```bash
+    OLD_VERSION=$(yq -r '.version' charts/plugin-barman-cloud/Chart.yaml)
+    OLD_APP_VERSION=$(yq -r '.appVersion' charts/plugin-barman-cloud/Chart.yaml)
+    echo Old chart version: $OLD_VERSION
+    echo Old app version: $OLD_APP_VERSION
+    ```
+2. Decide which version to create, depending on the kind of jump of the CloudNativePG release, following semver
+    semantics. For this document, let's call it `X.Y.Z`
+    ```bash
+    NEW_VERSION="X.Y.Z"
+    ```
+3. Create a branch named `release/plugin-barman-cloud-vX.Y.Z` and switch to it:
+    ```bash
+    git switch --create release/plugin-barman-cloud-v$NEW_VERSION
+    ```
+4. Update the `.version` in the [Chart.yaml](./charts/plugin-barman-cloud/Chart.yaml) file to `"X.Y.Z"`
+    ```bash
+    sed -i -E "s/^version: \"([0-9]+.?)+\"/version: \"$NEW_VERSION\"/" charts/plugin-barman-cloud/Chart.yaml
+    ```
+5. Update everything else as required, e.g. if releasing due to a new `plugin-barman-cloud` version being released, you might
+    want to:
+    1. Find the latest `plugin-barman-cloud` version by running:
+        ```bash
+        NEW_APP_VERSION=$(curl -Ssl "https://api.github.com/repos/cloudnative-pg/plugin-barman-cloud/tags" | jq -r '.[0].name')
+        echo New app version: $NEW_APP_VERSION
+        ```
+    2. Update `.appVersion` in the [Chart.yaml](./charts/plugin-barman-cloud/Chart.yaml) file
+        ```bash
+        sed -i -E "s/^appVersion: \"v([0-9]+.?)+\"/appVersion: \"$NEW_APP_VERSION\"/" charts/plugin-barman-cloud/Chart.yaml
+        ```
+    3. Update [crds.yaml](./charts/plugin-barman-cloud/templates/crds/crds.yaml), which can be built using
+        [kustomize](https://kustomize.io/) from the `plugin-barman-cloud` repo using kustomize
+        [remoteBuild](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md)
+        running:
+
+        Verify the version is correct. Edit it if incorrect, then run:
+        ```bash
+        echo '{{- if .Values.crds.create }}' > ./charts/plugin-barman-cloud/templates/crds/crds.yaml
+        kustomize build https://github.com/cloudnative-pg/plugin-barman-cloud/config/crd/\?ref\=$NEW_APP_VERSION >> ./charts/plugin-barman-cloud/templates/crds/crds.yaml
+        echo '{{- end }}' >> ./charts/plugin-barman-cloud/templates/crds/crds.yaml
+        ```
+
+        Check that the `helm.sh/resource-policy: keep` annotation is still present after regenerating the CRDs.
+    4. To update the files in the [templates](./charts/plugin-barman-cloud/templates) directory, you can diff the previous
+        CNPG release yaml against the new one, to find what should be updated (e.g.
+        ```bash
+        vimdiff \
+            "https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/${OLD_APP_VERSION}/manifest.yaml" \
+            "https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/${NEW_APP_VERSION}/manifest.yaml"
+       ```
+
+    5. Update [values.yaml](./charts/plugin-barman-cloud/values.yaml) if needed
+    6. NOTE: updating `values.yaml` just for the appVersion may not be necessary, as the value should default to the
+        `appVersion` in `Chart.yaml`
+6. Run `make docs schema` to regenerate the docs and the values schema in case it is needed
+    ```bash
+    make docs schema
+    ```
+7. Commit and add the relevant information you wish in the commit message.
+    ```bash
+    git add .
+    git commit -S -s -m "Release plugin-barman-cloud-v$NEW_VERSION" --edit
+    ```
+8. Push the new branch
+    ```bash
+    git push --set-upstream origin release/plugin-barman-cloud-v$NEW_VERSION
+    ```
+9. A PR named `Release plugin-barman-cloud-vX.Y.Z` should be automatically created
+10. Wait for all the checks to pass
+11. Two approvals are required in order to merge the PR, if you are a maintainer approve the PR yourself and ask for
+    another approval, otherwise ask for two approvals directly.
+12. Merge the PR squashing all commits and **taking care to keep the commit message to be
+    `Release plugin-barman-cloud-vX.Y.Z`**
+13. A release `plugin-barman-cloud-vX.Y.Z` should be automatically created by an action, which will then trigger the release
+    action. Verify they both are successful.
+14. Once done you should be able to run:
+    ```bash
+    helm repo add cnpg https://cloudnative-pg.github.io/charts
+    helm repo update
+    helm search repo cnpg
+    ```
+    and be able to see the new version `X.Y.Z` as `CHART VERSION` for `plugin-barman-cloud`

SECURITY.md

@@ -0,0 +1,4 @@
+# Security Policy
+
+For the most current and detailed version of our security procedures, please
+refer to the [official CloudNativePG Security Policy](https://github.com/cloudnative-pg/cloudnative-pg/blob/main/SECURITY.md).

charts/cloudnative-pg/templates/deployment.yaml

@@ -101,7 +101,7 @@ spec:
         livenessProbe:
           httpGet:
             path: /readyz
-            port: {{ .Values.webhook.port }}
+            port: webhook-server
             scheme: HTTPS
           {{- if .Values.webhook.livenessProbe.initialDelaySeconds }}
           initialDelaySeconds: {{ .Values.webhook.livenessProbe.initialDelaySeconds }}
@@ -117,7 +117,7 @@ spec:
         readinessProbe:
           httpGet:
             path: /readyz
-            port: {{ .Values.webhook.port }}
+            port: webhook-server
             scheme: HTTPS
           {{- if .Values.webhook.readinessProbe.initialDelaySeconds }}
           initialDelaySeconds: {{ .Values.webhook.readinessProbe.initialDelaySeconds }}
@@ -132,7 +132,7 @@ spec:
           {{- end }}
           httpGet:
             path: /readyz
-            port: {{ .Values.webhook.port }}
+            port: webhook-server
             scheme: HTTPS
           {{- if .Values.webhook.startupProbe.periodSeconds }}
           periodSeconds: {{ .Values.webhook.startupProbe.periodSeconds }}

charts/cloudnative-pg/values.yaml

@@ -210,30 +210,30 @@ monitoringQueriesConfigMap:
   queries: |
     backends:
       query: |
-       SELECT sa.datname
-           , sa.usename
-           , sa.application_name
-           , states.state
-           , COALESCE(sa.count, 0) AS total
-           , COALESCE(sa.max_tx_secs, 0) AS max_tx_duration_seconds
-           FROM ( VALUES ('active')
-               , ('idle')
-               , ('idle in transaction')
-               , ('idle in transaction (aborted)')
-               , ('fastpath function call')
-               , ('disabled')
-               ) AS states(state)
-           LEFT JOIN (
-               SELECT datname
-                   , state
-                   , usename
-                   , COALESCE(application_name, '') AS application_name
-                   , COUNT(*)
-                   , COALESCE(EXTRACT (EPOCH FROM (max(now() - xact_start))), 0) AS max_tx_secs
-               FROM pg_catalog.pg_stat_activity
-               GROUP BY datname, state, usename, application_name
-           ) sa ON states.state = sa.state
-           WHERE sa.usename IS NOT NULL
+        SELECT sa.datname
+          , sa.usename
+          , sa.application_name
+          , states.state
+          , COALESCE(sa.count, 0) AS total
+          , COALESCE(sa.max_tx_secs, 0) AS max_tx_duration_seconds
+        FROM ( VALUES ('active')
+          , ('idle')
+          , ('idle in transaction')
+          , ('idle in transaction (aborted)')
+          , ('fastpath function call')
+          , ('disabled')
+          ) AS states(state)
+        LEFT JOIN (
+          SELECT datname
+            , state
+            , usename
+            , COALESCE(application_name, '') AS application_name
+            , COUNT(*)
+            , COALESCE(EXTRACT (EPOCH FROM (max(now() - xact_start))), 0) AS max_tx_secs
+          FROM pg_catalog.pg_stat_activity
+          GROUP BY datname, state, usename, application_name
+        ) sa ON states.state = sa.state
+        WHERE sa.usename IS NOT NULL
       metrics:
         - datname:
             usage: "LABEL"
@@ -256,22 +256,22 @@ monitoringQueriesConfigMap:
 
     backends_waiting:
       query: |
-       SELECT count(*) AS total
-       FROM pg_catalog.pg_locks blocked_locks
-       JOIN pg_catalog.pg_locks blocking_locks
-         ON blocking_locks.locktype = blocked_locks.locktype
-         AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
-         AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
-         AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
-         AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
-         AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
-         AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
-         AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
-         AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
-         AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
-         AND blocking_locks.pid != blocked_locks.pid
-       JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
-       WHERE NOT blocked_locks.granted
+        SELECT count(*) AS total
+        FROM pg_catalog.pg_locks blocked_locks
+        JOIN pg_catalog.pg_locks blocking_locks
+          ON blocking_locks.locktype = blocked_locks.locktype
+          AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
+          AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
+          AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
+          AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
+          AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
+          AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
+          AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
+          AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
+          AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
+          AND blocking_locks.pid != blocked_locks.pid
+        JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
+        WHERE NOT blocked_locks.granted
       metrics:
         - total:
             usage: "GAUGE"
@@ -309,16 +309,17 @@ monitoringQueriesConfigMap:
             description: "Time at which postgres started (based on epoch)"
 
     pg_replication:
-      query: "SELECT CASE WHEN (
-                NOT pg_catalog.pg_is_in_recovery()
-                OR pg_catalog.pg_last_wal_receive_lsn() = pg_catalog.pg_last_wal_replay_lsn())
-              THEN 0
-              ELSE GREATEST (0,
-                EXTRACT(EPOCH FROM (now() - pg_catalog.pg_last_xact_replay_timestamp())))
-              END AS lag,
-              pg_catalog.pg_is_in_recovery() AS in_recovery,
-              EXISTS (TABLE pg_stat_wal_receiver) AS is_wal_receiver_up,
-              (SELECT count(*) FROM pg_catalog.pg_stat_replication) AS streaming_replicas"
+      query: |
+        SELECT CASE WHEN (
+            NOT pg_catalog.pg_is_in_recovery()
+            OR pg_catalog.pg_last_wal_receive_lsn() = pg_catalog.pg_last_wal_replay_lsn())
+          THEN 0
+          ELSE GREATEST (0,
+            EXTRACT(EPOCH FROM (now() - pg_catalog.pg_last_xact_replay_timestamp())))
+          END AS lag,
+          pg_catalog.pg_is_in_recovery() AS in_recovery,
+          EXISTS (TABLE pg_stat_wal_receiver) AS is_wal_receiver_up,
+          (SELECT count(*) FROM pg_catalog.pg_stat_replication) AS streaming_replicas
       metrics:
         - lag:
             usage: "GAUGE"
@@ -586,20 +587,20 @@ monitoringQueriesConfigMap:
     pg_stat_replication:
       primary: true
       query: |
-       SELECT usename
-         , COALESCE(application_name, '') AS application_name
-         , COALESCE(client_addr::text, '') AS client_addr
-         , COALESCE(client_port::text, '') AS client_port
-         , EXTRACT(EPOCH FROM backend_start) AS backend_start
-         , COALESCE(pg_catalog.age(backend_xmin), 0) AS backend_xmin_age
-         , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), sent_lsn) AS sent_diff_bytes
-         , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), write_lsn) AS write_diff_bytes
-         , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), flush_lsn) AS flush_diff_bytes
-         , COALESCE(pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), replay_lsn),0) AS replay_diff_bytes
-         , COALESCE((EXTRACT(EPOCH FROM write_lag)),0)::float AS write_lag_seconds
-         , COALESCE((EXTRACT(EPOCH FROM flush_lag)),0)::float AS flush_lag_seconds
-         , COALESCE((EXTRACT(EPOCH FROM replay_lag)),0)::float AS replay_lag_seconds
-       FROM pg_catalog.pg_stat_replication
+        SELECT usename
+          , COALESCE(application_name, '') AS application_name
+          , COALESCE(client_addr::text, '') AS client_addr
+          , COALESCE(client_port::text, '') AS client_port
+          , EXTRACT(EPOCH FROM backend_start) AS backend_start
+          , COALESCE(pg_catalog.age(backend_xmin), 0) AS backend_xmin_age
+          , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), sent_lsn) AS sent_diff_bytes
+          , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), write_lsn) AS write_diff_bytes
+          , pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), flush_lsn) AS flush_diff_bytes
+          , COALESCE(pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), replay_lsn),0) AS replay_diff_bytes
+          , COALESCE((EXTRACT(EPOCH FROM write_lag)),0)::float AS write_lag_seconds
+          , COALESCE((EXTRACT(EPOCH FROM flush_lag)),0)::float AS flush_lag_seconds
+          , COALESCE((EXTRACT(EPOCH FROM replay_lag)),0)::float AS replay_lag_seconds
+        FROM pg_catalog.pg_stat_replication
       metrics:
         - usename:
             usage: "LABEL"
@@ -659,13 +660,13 @@ monitoringQueriesConfigMap:
     pg_extensions:
       query: |
         SELECT
-         current_database() as datname,
-         name as extname,
-         default_version,
-         installed_version,
-         CASE
-           WHEN default_version = installed_version THEN 0
-           ELSE 1
+          current_database() as datname,
+          name as extname,
+          default_version,
+          installed_version,
+          CASE
+            WHEN default_version = installed_version THEN 0
+            ELSE 1
         END AS update_available
         FROM pg_catalog.pg_available_extensions
         WHERE installed_version IS NOT NULL

charts/cluster/README.md

@@ -168,6 +168,7 @@ refer to  the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
 | cluster.monitoring.customQueriesSecret | list | `[]` | The list of secrets containing the custom queries |
 | cluster.monitoring.disableDefaultQueries | bool | `false` | Whether the default queries should be injected. Set it to true if you don't want to inject default queries into the cluster. |
 | cluster.monitoring.enabled | bool | `false` | Whether to enable monitoring |
+| cluster.monitoring.instrumentation.logicalReplication | bool | `true` | Enable logical replication metrics |
 | cluster.monitoring.podMonitor.enabled | bool | `true` | Whether to enable the PodMonitor |
 | cluster.monitoring.podMonitor.metricRelabelings | list | `[]` | The list of metric relabelings for the PodMonitor. Applied to samples before ingestion. |
 | cluster.monitoring.podMonitor.relabelings | list | `[]` | The list of relabelings for the PodMonitor. Applied to samples before scraping. |