[DEFECT] Missing POD in multi-master context for first master node

Issue template is used for reporting defects or support issues.

There is no POD on the "FIRST" master node due to lack of toleration specification

Detailed Description

i am experiencing this issue that seems to occur specificly in a multi-master environment.

I deployed a multi-master cluster with 3 master nodes and 4 worker nodes.
2 VMs using haproxy+keepalived share a VIP address to redirect all kubernetes API calls traffic to one of the 3 master nodes behind the haproxy nodes.

During the deployment of the kubernetes cluster, it's important to note it was done on 3 phases:

• The first "master" node on which the "kubeadm init" command is typed
• The 2 other "master" nodes on which "kubeadm join --control-plane" command is typed
• The 4 worker nodes on which "kubeadm join" command is typed

During installation of trousseau, when applying daemonset manifest i noticed i had only 2 PODS deploying instead of 3.

After some investigations, it seems the FIRST master (the one on which "kubeadm init" was typed) have an additional taint "" that the 2 other control-plane nodes (those on which "kubeadm join --control-plane" was typed) don't have. It seaems that with this additional taint, the "FIRST" master doesn't match the daemonset tolerations criterions and no POD is deployed on this node.

There are 3 problems ordered by gravity with this:

• the "FIRST" master node without any kms-vault pod running on it is not able to encrypt secrets using the KMS ==> only 2/3 nodes can encrypt secrets: this gives a false "secured" feeling.
• the "FIRST" master node without any kms-vault pod running on it is not able to decrypt secrets encrypted by other nodes ==> Throws errors if api calls is grabbed by the node without kms-vault pod running
• the kube-api server crash after a certain time due to no possible communication with the local kms pod using the socket file ==> The kube-api crash after a lots of errors (see below).

Expected Behavior

A POD is depoyed on every master/control-plane nodes.

Current Behavior

A POD is not depoyed on the "FIRST" master/control-plane node.

Steps to Reproduce

1. Deploy a first master node using "kubeadm init" command
2. Deploy additional master nodes using "kubeadm join" command with "--control-plane" flag
3. Deploy worker nodes "kubeadm join"
4. Deploy Vault and Trousseeau using the methode described in https://github.com/ondat/trousseau/wiki/Trousseau-Deployment#

Context (Environment)

Possible Solution/Implementation

Add an additional toleration:

sysadmin@nisdevkubeadmin01:~/trousseau$ diff trousseau-daemonset.yaml trousseau-daemonset.yaml.ORIGINAL
112,114d111
<         - key: node-role.kubernetes.io/master
<           operator: Exists
<           effect: NoSchedule

Apply it:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl apply -f trousseau-daemonset.yaml
daemonset.apps/vault-kms-provider created

The daemon set now has a value of 3:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe daemonset.apps/vault-kms-provider -n kube-system
Name:           vault-kms-provider
Selector:       name=vault-kms-provider
Node-Selector:  <none>
Labels:         app=vault-kms-provider
                tier=control-plane
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
[...etc...]
Events:
  Type    Reason            Age        From                  Message
  ----    ------            ----       ----                  -------
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-qnkbc
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-klsmf
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-rwl8s

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get daemonset/vault-kms-provider -n kube-system
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vault-kms-provider   3         3         3       3            3           <none>          19s

Possible PR

---

Logs and elements regarding issue

The cluster nodes:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get nodes -o wide
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
nisdevkubemaster01   Ready    control-plane   47d   v1.25.0   10.2.10.11    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubemaster02   Ready    control-plane   47d   v1.25.0   10.2.10.12    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubemaster03   Ready    control-plane   47d   v1.25.0   10.2.10.13    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker01   Ready    <none>          47d   v1.25.0   10.2.10.21    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker02   Ready    <none>          47d   v1.25.0   10.2.10.22    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker03   Ready    <none>          47d   v1.25.0   10.2.10.23    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker04   Ready    <none>          47d   v1.25.0   10.2.10.24    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2

The etcd and kube-api pods

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l component=etcd -n kube-system
NAME                      READY   STATUS    RESTARTS      AGE
etcd-nisdevkubemaster01   1/1     Running   2 (51m ago)   47d
etcd-nisdevkubemaster02   1/1     Running   5 (11d ago)   47d
etcd-nisdevkubemaster03   1/1     Running   5 (11d ago)   47d


sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l component=kube-apiserver -n kube-system
NAME                                READY   STATUS    RESTARTS   AGE
kube-apiserver-nisdevkubemaster01   1/1     Running   0          15s
kube-apiserver-nisdevkubemaster02   1/1     Running   0          116s
kube-apiserver-nisdevkubemaster03   1/1     Running   0          108s

The specs of the daemonset:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe daemonset.apps/vault-kms-provider -n kube-system
Name:           vault-kms-provider
Selector:       name=vault-kms-provider
Node-Selector:  <none>
Labels:         app=vault-kms-provider
                tier=control-plane
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           name=vault-kms-provider
  Service Account:  vault-auth
  Init Containers:
   vault-agent:
    Image:      vault
    Port:       <none>
    Host Port:  <none>
    Args:
      agent
      -config=/etc/vault/vault-agent-config.hcl
      -log-level=debug
    Environment:
      VAULT_ADDR:  http://10.109.173.136:8200
    Mounts:
      /etc/secrets from shared-data (rw)
      /etc/vault from config (rw)
  Containers:
   vault-kms-provider:
    Image:      ghcr.io/ondat/trousseau:v1.1.3
    Port:       8787/TCP
    Host Port:  8787/TCP
    Args:
      -v=5
      --config-file-path=/opt/trousseau/config.yaml
      --listen-addr=unix:///opt/vault-kms/vaultkms.socket
      --zap-encoder=json
      --v=3
    Limits:
      cpu:     300m
      memory:  256Mi
    Requests:
      cpu:     50m
      memory:  64Mi
    Liveness:  http-get http://:8787/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      VAULT_SKIP_VERIFY:  true
    Mounts:
      /opt/trousseau/ from shared-data (rw)
      /opt/vault-kms from vault-kms (rw)
  Volumes:
   vault-kms:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/vault-kms
    HostPathType:  DirectoryOrCreate
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trousseau-vault-agent-config
    Optional:  false
   shared-data:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Events:
  Type    Reason            Age        From                  Message
  ----    ------            ----       ----                  -------
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-l7zpj
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-m27bd

The POD deployed:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l name=vault-kms-provider  -o wide -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
vault-kms-provider-l7zpj   1/1     Running   0          58s   10.2.10.13   nisdevkubemaster03   <none>           <none>
vault-kms-provider-m27bd   1/1     Running   0          58s   10.2.10.12   nisdevkubemaster02   <none>           <none>

The Daemonset states up of 2 desired POD instead of 3:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get daemonset/vault-kms-provider -n kube-system
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vault-kms-provider   2         2         2       2            2           <none>          2m10s

If encrypting a new secret

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl create secret generic secret-post-deploy -n default --from-literal=mykey=mydata
secret/secret-post-deploy created

It's retrieval can fail depending on which nodes handle the API request regarding the secret:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
NAME                 TYPE     DATA   AGE
secret-post-deploy   Opaque   1      2m51s


sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
Error from server: illegal base64 data at input byte 3

The error occurs on the node with encryption not activated.

The "FIRST" master throws same kind of errors regarding "illegal base64 data"

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs 87d0b2655ed7
I0515 17:35:34.072788       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:35:34.074047       1 server.go:161] Version: v1.25.0
I0515 17:35:34.074108       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0515 17:35:34.762893       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:35:34.764786       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:34.764810       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:35:34.766571       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:34.766590       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:35:34.835864       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:35:34.837134       1 instance.go:261] Using reconciler: lease
W0515 17:35:34.945556       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:34.945617       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:35.154092       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:35:35.689750       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.691989       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.697785       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:35:35.703823       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:35:35.706087       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.709230       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.709355       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.726067       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.726116       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.732634       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.732693       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.732877       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:35:35.745284       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.745356       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.748957       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.749067       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.758514       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.767660       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.775217       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:35:35.775259       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:35:35.777914       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.780079       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:35:35.781520       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:35.781540       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:35:35.808688       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.952375       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:35.952451       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:36.959224       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:36.959343       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:37.458302       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:35:37.458303       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:35:37.458766       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:35:37.458849       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:35:37.459986       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:35:37.460042       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:35:37.460118       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:35:37.466501       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:35:37.466535       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:35:37.466572       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:35:37.466577       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:35:37.467258       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:35:37.475120       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:35:37.475328       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:35:37.476453       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:35:37.476525       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:35:37.476595       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:35:37.476852       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:35:37.476892       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
E0515 17:35:37.513790       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:37.515436       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:37.515504       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
I0515 17:35:37.476977       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:35:37.479710       1 controller.go:85] Starting OpenAPI controller
I0515 17:35:37.479744       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:35:37.479771       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:35:37.479799       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:35:37.479818       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:35:37.479833       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:35:37.479850       1 crd_finalizer.go:266] Starting CRDFinalizer
I0515 17:35:37.486404       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:35:37.524909       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
I0515 17:35:37.560203       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:35:37.563100       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:35:37.566703       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:35:37.567239       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:35:37.578330       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:35:37.578482       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:35:37.628420       1 shared_informer.go:262] Caches are synced for crd-autoregister
W0515 17:35:37.964107       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:37.964178       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:38.114255       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0515 17:35:38.487235       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
E0515 17:35:38.579921       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:38.580766       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:38.580906       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:38.967167       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:38.967221       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:39.970543       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:39.970573       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
E0515 17:35:40.739566       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:40.740581       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:40.740638       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:40.975008       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:40.975578       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:41.979366       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:41.979466       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:42.983733       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:42.983839       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:43.987062       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:43.987161       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:44.990256       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:44.990284       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
E0515 17:35:45.729395       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:45.730317       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:45.730483       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:45.993650       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
[...etc...]

When activating the encryption on the "FIRST" master using the following manifest, new errors are thrown:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.2.10.11:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
[...]
    - --encryption-provider-config=/etc/kubernetes/custom/encryption-config.yaml
    - mountPath: /etc/kubernetes/custom
      name: custom-config
      readOnly: true
    - mountPath: /opt/vault-kms/
      name: kms-socket
      readOnly: false
[...]
  volumes:
[...]
  - hostPath:
      path: /etc/kubernetes/custom
      type: DirectoryOrCreate
    name: custom-config
[...]
  - hostPath:
      path: /opt/vault-kms
      type: DirectoryOrCreate
    name: kms-socket

The new errors thrown refers to issue with communication with gRPC server using socket file :

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs cc3db8189c01
I0515 17:42:16.157636       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:42:16.158614       1 server.go:161] Version: v1.25.0
I0515 17:42:16.158667       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E0515 17:42:16.646681       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:16.646889       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:16.647202       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:16.647293       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:16.648287       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:42:16.650018       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:16.650041       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:42:16.651554       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:16.651571       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:42:16.725046       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:42:16.727641       1 instance.go:261] Using reconciler: lease
I0515 17:42:17.029848       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:42:17.551934       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.554766       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.560536       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:42:17.566833       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:42:17.569340       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.573850       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.574112       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.584711       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.584743       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.592227       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.592266       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.592552       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:42:17.608699       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.608740       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.611583       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.611617       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.618695       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.625013       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.631577       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:42:17.631614       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:42:17.634346       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.636618       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:42:17.638131       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:17.638157       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
E0515 17:42:17.647716       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:17.647870       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
W0515 17:42:17.665928       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
I0515 17:42:19.357654       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:42:19.357873       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:42:19.376162       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:42:19.376807       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:42:19.376902       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:42:19.378304       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:42:19.378355       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:42:19.378884       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:42:19.381143       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:42:19.390073       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:42:19.390334       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:42:19.390808       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:42:19.390827       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:42:19.390862       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:42:19.390906       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:42:19.390918       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:42:19.390956       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:42:19.390966       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:42:19.394609       1 controller.go:85] Starting OpenAPI controller
I0515 17:42:19.394675       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:42:19.394701       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:42:19.394751       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:42:19.394774       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:42:19.394800       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:42:19.394887       1 crd_finalizer.go:266] Starting CRDFinalizer
I0515 17:42:19.395282       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:42:19.399703       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:42:19.401355       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:42:19.401373       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
E0515 17:42:19.431444       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:19.431662       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:19.448569       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:42:19.479595       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:42:19.490948       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:42:19.491143       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:42:19.491742       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:42:19.491781       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:42:19.506631       1 shared_informer.go:262] Caches are synced for crd-autoregister
E0515 17:42:19.549355       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:19.549582       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:19.808223       1 trace.go:205] Trace[1443041845]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:16.797) (total time: 3010ms):
Trace[1443041845]: [3.010156003s] [3.010156003s] END
W0515 17:42:19.808574       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:19.808662       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:20.392012       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
I0515 17:42:20.547441       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
E0515 17:42:21.202436       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:21.202563       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:21.856333       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:21.856433       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:22.421060       1 trace.go:205] Trace[995690186]: "List(recursive=true) etcd3" audit-id:1fc8fb24-bb4b-4409-a56f-fa7b2e9b10db,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:19.395) (total time: 3025ms):
Trace[995690186]: [3.025995675s] [3.025995675s] END
I0515 17:42:22.421257       1 trace.go:205] Trace[730370234]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:1fc8fb24-bb4b-4409-a56f-fa7b2e9b10db,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:19.394) (total time: 3026ms):
Trace[730370234]: [3.026263657s] [3.026263657s] END
W0515 17:42:22.422844       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:22.423190       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:23.815578       1 trace.go:205] Trace[1183215355]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:20.809) (total time: 3005ms):
Trace[1183215355]: [3.005657272s] [3.005657272s] END
W0515 17:42:23.815635       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:23.815679       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:25.436566       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:25.436634       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:25.526406       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:25.526592       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:26.863508       1 trace.go:205] Trace[541467408]: "List(recursive=true) etcd3" audit-id:31c9c553-aa87-495f-a05e-a7524e7a4948,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:23.858) (total time: 3005ms):
Trace[541467408]: [3.005291123s] [3.005291123s] END
I0515 17:42:26.863998       1 trace.go:205] Trace[914437271]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:31c9c553-aa87-495f-a05e-a7524e7a4948,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:23.858) (total time: 3005ms):
Trace[914437271]: [3.005834219s] [3.005834219s] END
W0515 17:42:26.864788       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:26.865131       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:27.818852       1 trace.go:205] Trace[2013110149]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:24.816) (total time: 3002ms):
Trace[2013110149]: [3.002695001s] [3.002695001s] END
W0515 17:42:27.818903       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:27.818920       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:30.910587       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:30.910938       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:31.437922       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:31.437987       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:31.823964       1 trace.go:205] Trace[1637434811]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:28.819) (total time: 3004ms):
Trace[1637434811]: [3.004718217s] [3.004718217s] END
W0515 17:42:31.824006       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:31.824100       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:32.310942       1 trace.go:205] Trace[1765368075]: "List(recursive=true) etcd3" audit-id:892c4e2c-8519-468a-9d33-c71f84e2ad79,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:29.308) (total time: 3002ms):
Trace[1765368075]: [3.002766092s] [3.002766092s] END
I0515 17:42:32.311074       1 trace.go:205] Trace[860303514]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:892c4e2c-8519-468a-9d33-c71f84e2ad79,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:29.308) (total time: 3002ms):
Trace[860303514]: [3.002944476s] [3.002944476s] END
W0515 17:42:32.311594       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:32.311623       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:35.827724       1 trace.go:205] Trace[5852664]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:32.824) (total time: 3002ms):
Trace[5852664]: [3.002953826s] [3.002953826s] END
W0515 17:42:35.827852       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:35.827889       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:38.011513       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:38.011628       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:39.833073       1 trace.go:205] Trace[313620331]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:36.828) (total time: 3004ms):
Trace[313620331]: [3.004095391s] [3.004095391s] END
W0515 17:42:39.833220       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:39.833263       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:39.946339       1 trace.go:205] Trace[160050274]: "List(recursive=true) etcd3" audit-id:ee5c6eae-099a-4d90-95ea-4716bcaead40,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:36.943) (total time: 3002ms):
Trace[160050274]: [3.002653112s] [3.002653112s] END
I0515 17:42:39.946610       1 trace.go:205] Trace[2133478831]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ee5c6eae-099a-4d90-95ea-4716bcaead40,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:36.943) (total time: 3002ms):
Trace[2133478831]: [3.002964763s] [3.002964763s] END
W0515 17:42:39.947143       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:39.947177       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
[...etc...]

Furthermore retrieval of secrets still generated issue (regarding the socket communication):

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
Error from server (InternalError): Internal error occurred: failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"

If we look more at node description, we can notice the "FIRST" master avec 2 Taints: "node-role.kubernetes.io/control-plane:NoSchedule" and "node-role.kubernetes.io/master:NoSchedule".

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster01
Name:               nisdevkubemaster01
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 18:59:03 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
                    node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false



sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster02
Name:               nisdevkubemaster02
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster02
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 19:00:31 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false



sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster03
Name:               nisdevkubemaster03
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster03
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 19:00:30 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false

After a certain time, the "FIRST" master kube-apiserver POD is exited and restarted

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker ps -a | grep -i kube-api
80e92655e694   4d2edfd10d3e                "kube-apiserver --ad…"   2 minutes ago       Up 2 minutes                             k8s_kube-apiserver_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_2
0ff23e9cd858   4d2edfd10d3e                "kube-apiserver --ad…"   6 minutes ago       Exited (137) 2 minutes ago               k8s_kube-apiserver_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_1
74d59a631d86   registry.k8s.io/pause:3.6   "/pause"                 11 minutes ago      Up 11 minutes                            k8s_POD_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_0



root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs 0ff23e9cd858
I0515 17:46:56.058530       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:46:56.059182       1 server.go:161] Version: v1.25.0
I0515 17:46:56.059246       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E0515 17:46:56.782143       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
E0515 17:46:56.782147       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:56.782243       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
W0515 17:46:56.782261       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:56.782341       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:46:56.782768       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:56.782780       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:46:56.784095       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:56.784119       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:46:56.840202       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:46:56.841432       1 instance.go:261] Using reconciler: lease
I0515 17:46:57.089775       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:46:57.536347       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.544060       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.568767       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:46:57.574544       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:46:57.576934       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.579022       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.579086       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.583644       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.583664       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.585327       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.585343       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.585390       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:46:57.589827       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.589850       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.591475       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.591495       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.596328       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.600822       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.606039       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:46:57.606061       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:46:57.607975       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.609603       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:46:57.610697       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:57.610712       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:46:57.637286       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
E0515 17:46:57.783174       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:57.783245       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:46:59.104994       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:59.105093       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:59.159717       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:46:59.159870       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:46:59.161782       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:46:59.164069       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:46:59.164202       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:46:59.165241       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:46:59.165397       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:46:59.165570       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:46:59.167261       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:46:59.167290       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:46:59.169569       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:46:59.169586       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
I0515 17:46:59.169809       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:46:59.189651       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:46:59.190583       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:46:59.193829       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:46:59.196657       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:46:59.196470       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:46:59.197268       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:46:59.197337       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:46:59.197659       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:46:59.198127       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:46:59.199915       1 controller.go:85] Starting OpenAPI controller
I0515 17:46:59.200020       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:46:59.200091       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:46:59.200160       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:46:59.200230       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:46:59.200265       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:46:59.200320       1 crd_finalizer.go:266] Starting CRDFinalizer
E0515 17:46:59.288402       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:59.288527       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:59.297543       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:46:59.297557       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:46:59.380140       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:46:59.381196       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:46:59.381355       1 shared_informer.go:262] Caches are synced for crd-autoregister
I0515 17:46:59.383050       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:46:59.386369       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:46:59.879843       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0515 17:46:59.902711       1 trace.go:205] Trace[41463727]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:46:56.897) (total time: 3005ms):
Trace[41463727]: [3.005633161s] [3.005633161s] END
W0515 17:46:59.903173       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:46:59.903355       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:00.167691       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
E0515 17:47:01.025758       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:01.025903       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:01.944001       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:01.944083       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:02.279646       1 trace.go:205] Trace[284735070]: "List(recursive=true) etcd3" audit-id:ba3219cb-566b-48b2-93ee-480bdef2ac06,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:46:59.270) (total time: 3008ms):
Trace[284735070]: [3.008960631s] [3.008960631s] END
I0515 17:47:02.280014       1 trace.go:205] Trace[2055471106]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ba3219cb-566b-48b2-93ee-480bdef2ac06,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:46:59.270) (total time: 3009ms):
Trace[2055471106]: [3.009334798s] [3.009334798s] END
W0515 17:47:02.280715       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:02.280840       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:03.907210       1 trace.go:205] Trace[2042703226]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:00.903) (total time: 3003ms):
Trace[2042703226]: [3.003285857s] [3.003285857s] END
W0515 17:47:03.907316       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:03.907546       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:05.296515       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:05.296698       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:06.161563       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:06.161636       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:06.526685       1 trace.go:205] Trace[730767454]: "List(recursive=true) etcd3" audit-id:ce524b37-6fed-4477-a9bf-dca49887e782,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:03.523) (total time: 3003ms):
Trace[730767454]: [3.003023214s] [3.003023214s] END
I0515 17:47:06.527254       1 trace.go:205] Trace[2058701975]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ce524b37-6fed-4477-a9bf-dca49887e782,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:03.523) (total time: 3003ms):
Trace[2058701975]: [3.003562365s] [3.003562365s] END
W0515 17:47:06.528416       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:06.528457       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:07.911678       1 trace.go:205] Trace[1192110976]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:04.908) (total time: 3003ms):
Trace[1192110976]: [3.003093004s] [3.003093004s] END
W0515 17:47:07.911821       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:07.911902       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:11.394880       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:11.395074       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:11.757641       1 trace.go:205] Trace[619589245]: "List(recursive=true) etcd3" audit-id:9fc0e54c-2bce-462f-8687-dd5bb4e8d2d4,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:08.754) (total time: 3003ms):
Trace[619589245]: [3.003093465s] [3.003093465s] END
I0515 17:47:11.757763       1 trace.go:205] Trace[745073040]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:9fc0e54c-2bce-462f-8687-dd5bb4e8d2d4,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:08.754) (total time: 3003ms):
Trace[745073040]: [3.003258404s] [3.003258404s] END
W0515 17:47:11.758294       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:11.758343       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:11.915617       1 trace.go:205] Trace[350332463]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:08.912) (total time: 3003ms):
Trace[350332463]: [3.003056785s] [3.003056785s] END
W0515 17:47:11.915661       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:11.915678       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:13.880230       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:13.880601       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:15.924355       1 trace.go:205] Trace[1803151140]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:12.916) (total time: 3007ms):
Trace[1803151140]: [3.007638796s] [3.007638796s] END
W0515 17:47:15.924471       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:15.924503       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:18.917633       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:18.917723       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:19.930171       1 trace.go:205] Trace[961899593]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:16.925) (total time: 3004ms):
Trace[961899593]: [3.00455736s] [3.00455736s] END
W0515 17:47:19.930262       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:19.930289       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:20.156403       1 trace.go:205] Trace[1041693117]: "List(recursive=true) etcd3" audit-id:a50c74de-05f0-47b6-9da5-0f62c9c3cbee,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:17.152) (total time: 3003ms):
Trace[1041693117]: [3.003846681s] [3.003846681s] END
I0515 17:47:20.156586       1 trace.go:205] Trace[1219087519]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:a50c74de-05f0-47b6-9da5-0f62c9c3cbee,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:17.152) (total time: 3004ms):
Trace[1219087519]: [3.00409859s] [3.00409859s] END
W0515 17:47:20.157050       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:20.157090       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:23.846723       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:23.846996       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:23.935202       1 trace.go:205] Trace[669387932]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:20.931) (total time: 3003ms):
Trace[669387932]: [3.003739883s] [3.003739883s] END
W0515 17:47:23.935259       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:23.935274       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:27.941235       1 trace.go:205] Trace[1458727689]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:24.935) (total time: 3005ms):
Trace[1458727689]: [3.005547556s] [3.005547556s] END
W0515 17:47:27.941283       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:27.941296       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:29.695725       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:29.695887       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:31.945679       1 trace.go:205] Trace[1144863289]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:28.941) (total time: 3004ms):
Trace[1144863289]: [3.004178676s] [3.004178676s] END
W0515 17:47:31.945732       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:31.945769       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:35.451608       1 trace.go:205] Trace[1834458678]: "List(recursive=true) etcd3" audit-id:6417048f-cd2f-4f68-ba11-2184f2845129,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:32.447) (total time: 3004ms):
Trace[1834458678]: [3.004312645s] [3.004312645s] END
I0515 17:47:35.451797       1 trace.go:205] Trace[73531258]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:6417048f-cd2f-4f68-ba11-2184f2845129,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:32.447) (total time: 3004ms):
Trace[73531258]: [3.004528771s] [3.004528771s] END
W0515 17:47:35.452286       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:35.452343       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:35.951913       1 trace.go:205] Trace[353802577]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:32.946) (total time: 3005ms):
Trace[353802577]: [3.005047047s] [3.005047047s] END
W0515 17:47:35.951972       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:35.951987       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:37.896794       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:37.896942       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:39.956952       1 trace.go:205] Trace[1720376311]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:36.952) (total time: 3003ms):
Trace[1720376311]: [3.00397061s] [3.00397061s] END
W0515 17:47:39.957001       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:39.957015       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:43.962075       1 trace.go:205] Trace[732546678]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:40.958) (total time: 3004ms):
Trace[732546678]: [3.004016272s] [3.004016272s] END
W0515 17:47:43.962138       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:43.962152       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:47.896908       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:47.897092       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:47.966712       1 trace.go:205] Trace[466044327]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:44.962) (total time: 3003ms):
Trace[466044327]: [3.003962077s] [3.003962077s] END
W0515 17:47:47.966871       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:47.966911       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:51.971314       1 trace.go:205] Trace[152847791]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:48.967) (total time: 3003ms):
Trace[152847791]: [3.00361396s] [3.00361396s] END
W0515 17:47:51.971366       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:51.971382       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:55.975350       1 trace.go:205] Trace[548900740]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:52.971) (total time: 3003ms):
Trace[548900740]: [3.003577628s] [3.003577628s] END
W0515 17:47:55.975507       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:55.975628       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:58.656782       1 trace.go:205] Trace[1204988067]: "List(recursive=true) etcd3" audit-id:7246c517-f979-46df-98f3-91e50385d9fc,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:55.652) (total time: 3003ms):
Trace[1204988067]: [3.003753233s] [3.003753233s] END
I0515 17:47:58.656919       1 trace.go:205] Trace[1807520032]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:7246c517-f979-46df-98f3-91e50385d9fc,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:55.652) (total time: 3003ms):
Trace[1807520032]: [3.003919313s] [3.003919313s] END
W0515 17:47:58.657616       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:58.657882       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:59.980923       1 trace.go:205] Trace[1666681437]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:56.976) (total time: 3004ms):
Trace[1666681437]: [3.004235707s] [3.004235707s] END
W0515 17:47:59.981023       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:59.981051       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:03.985857       1 trace.go:205] Trace[1780984614]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:00.981) (total time: 3004ms):
Trace[1780984614]: [3.004008418s] [3.004008418s] END
W0515 17:48:03.985952       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:03.985988       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:07.994368       1 trace.go:205] Trace[441807241]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:04.986) (total time: 3007ms):
Trace[441807241]: [3.007650779s] [3.007650779s] END
W0515 17:48:07.994425       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:07.994448       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:09.232762       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:09.232841       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:11.997734       1 trace.go:205] Trace[814431104]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:08.994) (total time: 3002ms):
Trace[814431104]: [3.002826481s] [3.002826481s] END
W0515 17:48:11.997779       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:11.997829       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:16.001887       1 trace.go:205] Trace[1968403247]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:12.997) (total time: 3003ms):
Trace[1968403247]: [3.003822832s] [3.003822832s] END
W0515 17:48:16.001954       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:16.002472       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:18.295272       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:18.295583       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:20.005099       1 trace.go:205] Trace[711726798]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:17.002) (total time: 3002ms):
Trace[711726798]: [3.00240867s] [3.00240867s] END
W0515 17:48:20.005195       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:20.005227       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:24.009309       1 trace.go:205] Trace[490107260]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:21.005) (total time: 3003ms):
Trace[490107260]: [3.003591429s] [3.003591429s] END
W0515 17:48:24.009363       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:24.009380       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:28.015116       1 trace.go:205] Trace[389252337]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:25.010) (total time: 3004ms):
Trace[389252337]: [3.004553925s] [3.004553925s] END
W0515 17:48:28.015160       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:28.015177       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:32.020501       1 trace.go:205] Trace[1750352782]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:29.016) (total time: 3004ms):
Trace[1750352782]: [3.004376719s] [3.004376719s] END
W0515 17:48:32.020554       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:32.020573       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:36.025053       1 trace.go:205] Trace[1873891737]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:33.021) (total time: 3003ms):
Trace[1873891737]: [3.003544018s] [3.003544018s] END
W0515 17:48:36.025110       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:36.025171       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:40.030964       1 trace.go:205] Trace[1978618791]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:37.026) (total time: 3004ms):
Trace[1978618791]: [3.004201314s] [3.004201314s] END
W0515 17:48:40.031018       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:40.031032       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:44.035805       1 trace.go:205] Trace[206298777]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:41.031) (total time: 3004ms):
Trace[206298777]: [3.004347769s] [3.004347769s] END
W0515 17:48:44.035854       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:44.035872       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:44.688191       1 trace.go:205] Trace[931530703]: "List(recursive=true) etcd3" audit-id:4d6105b8-65e3-4bb6-bd51-6c219c3945f2,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:48:41.685) (total time: 3002ms):
Trace[931530703]: [3.002950201s] [3.002950201s] END
I0515 17:48:44.688394       1 trace.go:205] Trace[812410954]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:4d6105b8-65e3-4bb6-bd51-6c219c3945f2,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:48:41.685) (total time: 3003ms):
Trace[812410954]: [3.003192572s] [3.003192572s] END
W0515 17:48:44.689163       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:44.689414       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:48.040105       1 trace.go:205] Trace[147757836]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:45.036) (total time: 3003ms):
Trace[147757836]: [3.003306505s] [3.003306505s] END
W0515 17:48:48.040228       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:48.040281       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:52.045261       1 trace.go:205] Trace[2037452132]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:49.040) (total time: 3004ms):
Trace[2037452132]: [3.004299925s] [3.004299925s] END
W0515 17:48:52.045317       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:52.045332       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:56.051162       1 trace.go:205] Trace[1123320100]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:53.045) (total time: 3005ms):
Trace[1123320100]: [3.005122395s] [3.005122395s] END
W0515 17:48:56.051249       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:56.051297       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:57.769721       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:57.769868       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:57.876917       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:57.877027       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:00.055653       1 trace.go:205] Trace[1096111447]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:57.051) (total time: 3003ms):
Trace[1096111447]: [3.003920014s] [3.003920014s] END
W0515 17:49:00.055788       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:00.055826       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:04.063245       1 trace.go:205] Trace[952464393]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:01.056) (total time: 3007ms):
Trace[952464393]: [3.007105596s] [3.007105596s] END
W0515 17:49:04.063315       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:04.063368       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:08.067562       1 trace.go:205] Trace[1379417391]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:05.064) (total time: 3003ms):
Trace[1379417391]: [3.003355436s] [3.003355436s] END
W0515 17:49:08.067650       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:08.067751       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:12.071789       1 trace.go:205] Trace[836301313]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:09.068) (total time: 3002ms):
Trace[836301313]: [3.002980546s] [3.002980546s] END
W0515 17:49:12.071841       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:12.071855       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:16.076365       1 trace.go:205] Trace[597938773]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:13.072) (total time: 3003ms):
Trace[597938773]: [3.003710619s] [3.003710619s] END
W0515 17:49:16.076414       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:16.076602       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:20.081414       1 trace.go:205] Trace[1701438819]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:17.077) (total time: 3003ms):
Trace[1701438819]: [3.003677276s] [3.003677276s] END
W0515 17:49:20.081487       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:20.081506       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:24.085053       1 trace.go:205] Trace[1036974030]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:21.082) (total time: 3002ms):
Trace[1036974030]: [3.002925632s] [3.002925632s] END
W0515 17:49:24.085092       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:24.085145       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:28.090134       1 trace.go:205] Trace[1187718898]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:25.085) (total time: 3004ms):
Trace[1187718898]: [3.004364001s] [3.004364001s] END
W0515 17:49:28.090422       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:28.090521       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:32.094174       1 trace.go:205] Trace[118669017]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:29.090) (total time: 3003ms):
Trace[118669017]: [3.003385054s] [3.003385054s] END
W0515 17:49:32.094232       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:32.094247       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:35.673699       1 trace.go:205] Trace[1825072249]: "List(recursive=true) etcd3" audit-id:7a1b9680-5ff4-4dc5-865f-604d15ab6dd4,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:49:32.669) (total time: 3003ms):
Trace[1825072249]: [3.003719034s] [3.003719034s] END
I0515 17:49:35.673817       1 trace.go:205] Trace[1409462059]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:7a1b9680-5ff4-4dc5-865f-604d15ab6dd4,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:49:32.669) (total time: 3003ms):
Trace[1409462059]: [3.003886146s] [3.003886146s] END
W0515 17:49:35.674398       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:35.674656       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:36.097032       1 trace.go:205] Trace[1958951863]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:33.094) (total time: 3002ms):
Trace[1958951863]: [3.002615148s] [3.002615148s] END
W0515 17:49:36.097070       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:36.097083       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:40.100709       1 trace.go:205] Trace[294960884]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:37.097) (total time: 3003ms):
Trace[294960884]: [3.003083128s] [3.003083128s] END
W0515 17:49:40.100758       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:40.100775       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:44.103671       1 trace.go:205] Trace[2104243303]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:41.101) (total time: 3002ms):
Trace[2104243303]: [3.002082077s] [3.002082077s] END
W0515 17:49:44.103797       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:44.103827       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:48.108662       1 trace.go:205] Trace[470392946]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:45.104) (total time: 3004ms):
Trace[470392946]: [3.004146033s] [3.004146033s] END
W0515 17:49:48.108737       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:48.108754       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:52.113106       1 trace.go:205] Trace[752840887]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:49.109) (total time: 3003ms):
Trace[752840887]: [3.003751281s] [3.003751281s] END
W0515 17:49:52.113157       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:52.113174       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:49:54.541978       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:49:54.542061       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:56.117416       1 trace.go:205] Trace[1281744931]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:53.113) (total time: 3003ms):
Trace[1281744931]: [3.003416608s] [3.003416608s] END
W0515 17:49:56.117517       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:56.117539       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:00.122283       1 trace.go:205] Trace[529705241]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:57.118) (total time: 3003ms):
Trace[529705241]: [3.003648513s] [3.003648513s] END
W0515 17:50:00.122336       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:00.122352       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:50:02.885240       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:50:02.885419       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:50:04.127379       1 trace.go:205] Trace[970261590]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:01.123) (total time: 3003ms):
Trace[970261590]: [3.003998927s] [3.003998927s] END
W0515 17:50:04.127480       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:04.127834       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:08.133833       1 trace.go:205] Trace[654433592]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:05.128) (total time: 3004ms):
Trace[654433592]: [3.004976274s] [3.004976274s] END
W0515 17:50:08.134082       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:08.134171       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:12.137953       1 trace.go:205] Trace[464761580]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:09.134) (total time: 3003ms):
Trace[464761580]: [3.003587332s] [3.003587332s] END
W0515 17:50:12.138000       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:12.138019       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:14.450160       1 trace.go:205] Trace[1262963168]: "Get" url:/api/v1/namespaces/default/secrets/secret-post-deploy,user-agent:kubectl/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:fd988e33-8946-4c60-855f-f523d8c0c547,client:10.2.10.3,accept:application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json,protocol:HTTP/2.0 (15-May-2023 17:50:11.447) (total time: 3002ms):
Trace[1262963168]: [3.002631314s] [3.002631314s] END
I0515 17:50:16.143390       1 trace.go:205] Trace[636923892]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:13.139) (total time: 3004ms):
Trace[636923892]: [3.004134024s] [3.004134024s] END
W0515 17:50:16.143474       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:16.143500       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:20.149658       1 trace.go:205] Trace[684083666]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:17.144) (total time: 3005ms):
Trace[684083666]: [3.005003758s] [3.005003758s] END
W0515 17:50:20.149738       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:20.149768       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:24.154513       1 trace.go:205] Trace[1678743422]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:21.150) (total time: 3004ms):
Trace[1678743422]: [3.004155949s] [3.004155949s] END
W0515 17:50:24.154586       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:24.154609       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:24.766317       1 trace.go:205] Trace[195294388]: "Get" url:/api/v1/namespaces/default/secrets/secret-post-deploy,user-agent:kubectl/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:aea0ae66-2ba2-4405-a3f0-4f3b180e70e1,client:10.2.10.3,accept:application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json,protocol:HTTP/2.0 (15-May-2023 17:50:21.762) (total time: 3003ms):
Trace[195294388]: [3.003868712s] [3.003868712s] END
I0515 17:50:28.159887       1 trace.go:205] Trace[1589446782]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:25.155) (total time: 3004ms):
Trace[1589446782]: [3.004832279s] [3.004832279s] END
W0515 17:50:28.159940       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:28.159959       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:28.260761       1 trace.go:205] Trace[1933186184]: "List(recursive=true) etcd3" audit-id:5197abbb-c335-4216-8938-6f0f002499aa,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:50:25.255) (total time: 3004ms):
Trace[1933186184]: [3.004832083s] [3.004832083s] END
I0515 17:50:28.261200       1 trace.go:205] Trace[2106224912]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:5197abbb-c335-4216-8938-6f0f002499aa,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:50:25.255) (total time: 3005ms):
Trace[2106224912]: [3.005344858s] [3.005344858s] END
W0515 17:50:28.262106       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:28.262155       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:50:32.165829       1 trace.go:205] Trace[344162878]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:29.161) (total time: 3004ms):
Trace[344162878]: [3.00459021s] [3.00459021s] END
W0515 17:50:32.165896       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:32.165968       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:36.170380       1 trace.go:205] Trace[355063030]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:33.166) (total time: 3004ms):
Trace[355063030]: [3.004035253s] [3.004035253s] END
W0515 17:50:36.170451       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:36.170528       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:40.175222       1 trace.go:205] Trace[1268436977]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:37.170) (total time: 3004ms):
Trace[1268436977]: [3.004226319s] [3.004226319s] END
W0515 17:50:40.175287       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:40.175306       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:44.180048       1 trace.go:205] Trace[1508039681]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:41.175) (total time: 3004ms):
Trace[1508039681]: [3.004068572s] [3.004068572s] END
W0515 17:50:44.180220       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:44.180268       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:48.185920       1 trace.go:205] Trace[1825445912]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:45.181) (total time: 3004ms):
Trace[1825445912]: [3.004743747s] [3.004743747s] END
W0515 17:50:48.186052       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:48.186079       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:52.192550       1 trace.go:205] Trace[16415966]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:49.187) (total time: 3004ms):
Trace[16415966]: [3.004834147s] [3.004834147s] END
W0515 17:50:52.192607       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:52.192626       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:55.747708       1 controller.go:211] Shutting down kubernetes service endpoint reconciler
E0515 17:50:55.751203       1 controller.go:214] Unable to remove endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.2.10.11, ResourceVersion: 0, AdditionalErrorMsg:
I0515 17:50:55.751604       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.751738       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.751779       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.753007       1 controller.go:115] Shutting down OpenAPI V3 controller
I0515 17:50:55.753185       1 controller.go:122] Shutting down OpenAPI controller
I0515 17:50:55.753272       1 apf_controller.go:309] Shutting down API Priority and Fairness config worker
I0515 17:50:55.753362       1 storage_flowcontrol.go:172] APF bootstrap ensurer is exiting
I0515 17:50:55.753449       1 crdregistration_controller.go:142] Shutting down crd-autoregister controller
I0515 17:50:55.753545       1 autoregister_controller.go:165] Shutting down autoregister controller
I0515 17:50:55.753628       1 apiservice_controller.go:131] Shutting down APIServiceRegistrationController
I0515 17:50:55.753707       1 nonstructuralschema_controller.go:204] Shutting down NonStructuralSchemaConditionController
I0515 17:50:55.753758       1 establishing_controller.go:87] Shutting down EstablishingController
I0515 17:50:55.753796       1 naming_controller.go:302] Shutting down NamingConditionController
I0515 17:50:55.753822       1 apiapproval_controller.go:198] Shutting down KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:50:55.753852       1 crd_finalizer.go:278] Shutting down CRDFinalizer
I0515 17:50:55.753887       1 customresource_discovery_controller.go:245] Shutting down DiscoveryController
I0515 17:50:55.753915       1 available_controller.go:503] Shutting down AvailableConditionController
I0515 17:50:55.754025       1 controller.go:89] Shutting down OpenAPI AggregationController
I0515 17:50:55.754046       1 controller.go:86] Shutting down OpenAPI V3 AggregationController
I0515 17:50:55.754081       1 dynamic_serving_content.go:146] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:50:55.754134       1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting"
I0515 17:50:55.754245       1 dynamic_cafile_content.go:171] "Shutting down controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:50:55.754525       1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller
I0515 17:50:55.754597       1 dynamic_cafile_content.go:171] "Shutting down controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:50:55.755394       1 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:50:55.755539       1 tlsconfig.go:255] "Shutting down DynamicServingCertificateController"
I0515 17:50:55.755493       1 secure_serving.go:255] Stopped listening on [::]:6443
I0515 17:50:55.755742       1 dynamic_serving_content.go:146] "Shutting down controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:50:55.754245       1 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:50:56.202330       1 trace.go:205] Trace[287869255]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:53.192) (total time: 3009ms):
Trace[287869255]: [3.009431743s] [3.009431743s] END
W0515 17:50:56.202405       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:56.202425       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:00.207279       1 trace.go:205] Trace[1949987037]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:57.202) (total time: 3004ms):
Trace[1949987037]: [3.004401066s] [3.004401066s] END
W0515 17:51:00.207355       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:00.207379       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:04.213154       1 trace.go:205] Trace[1488463345]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:01.208) (total time: 3004ms):
Trace[1488463345]: [3.004534501s] [3.004534501s] END
W0515 17:51:04.213229       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:04.213248       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:08.218396       1 trace.go:205] Trace[996780107]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:05.214) (total time: 3004ms):
Trace[996780107]: [3.00417563s] [3.00417563s] END
W0515 17:51:08.218468       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:08.218815       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:12.225289       1 trace.go:205] Trace[1359912236]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:09.219) (total time: 3005ms):
Trace[1359912236]: [3.005966479s] [3.005966479s] END
W0515 17:51:12.225726       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:12.225789       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:16.233065       1 trace.go:205] Trace[1939012164]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:13.227) (total time: 3005ms):
Trace[1939012164]: [3.005906292s] [3.005906292s] END
W0515 17:51:16.233189       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:16.233221       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:20.238277       1 trace.go:205] Trace[582931785]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:17.233) (total time: 3004ms):
Trace[582931785]: [3.004471313s] [3.004471313s] END
W0515 17:51:20.238355       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:20.238382       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:24.248342       1 trace.go:205] Trace[2105727503]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:21.238) (total time: 3009ms):
Trace[2105727503]: [3.009664569s] [3.009664569s] END
W0515 17:51:24.248418       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:24.248439       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...

[romdalf]

Thank you for your issue.

Supporting only KMSv1, the Trousseau project is no longer suitable for use except for educational purposes.
The Kubernetes project retired the KMSv1 in favor of KMSv2, a production-grade replacement. There is no plan (as far as I know) to develop a KMSv2-supported version.

There are other options available like:

• https://github.com/FalcoSuessgott/vault-kubernetes-kms
• https://github.com/beezy-dev/kleidi (I dev'ed this one with contributions from Red Hat/Nokia).

Cheers,
Rom

Note that I am not part of the Ondat org anymore, but I feel that I should provide the community with answers.