Unable to run kubera components on Konvoy

[shiveshabhishek]

Since Tuesday(yesterday) after the gitlab came up, when the Konvoy pipeline got triggered the Kubera components took more than expected time(more than 25-30 mins) and still pods weren't up.

[shiveshabhishek]

The above issue is happening on all the clusters of Konvoy (C1,C2 and C3).
Current status when tried to install kubera, status of kubera components:

konvoy-oep@rack2:~/shivesh$ kubectl get po 
NAME                                                    READY   STATUS             RESTARTS   AGE
alertmanager-6cb9896d58-hnc78                           0/1     Init:2/3           0          31m
alertstore-6df57df9dc-ct5wk                             0/1     Init:2/3           0          31m
alertstore-tablemanager-6d798c756d-p85ks                0/1     ImagePullBackOff   0          31m
cassandra-0                                             1/1     Running            0          31m
chat-server-9fd87877d-kw2r8                             0/1     ErrImagePull       0          31m
cloud-agent-759985db88-2ckg2                            1/1     Running            0          31m
configs-54865d6f6f-r8dfn                                1/1     Running            0          31m
configs-db-74d4fcd6c9-8bxjm                             1/1     Running            0          31m
consul-6c775c669-phf2t                                  1/1     Running            0          31m
distributor-75c86cd857-cqkn5                            0/1     Init:2/3           0          31m
elastalert-d8c884dbb-fs87r                              0/1     Init:1/2           0          31m
ingester-5794997bb-lrkj4                                0/1     PodInitializing    0          31m
kubera-nginx-ingress-controller-b59jn                   1/1     Running            0          31m
kubera-nginx-ingress-controller-cpk8q                   1/1     Running            0          31m
kubera-nginx-ingress-controller-d6wg4                   1/1     Running            0          31m
kubera-nginx-ingress-controller-hntbg                   1/1     Running            0          31m
kubera-nginx-ingress-controller-t7724                   1/1     Running            0          31m
kubera-nginx-ingress-default-backend-759497475f-rfbp2   1/1     Running            0          31m
kubera-nginx-ingress-default-backend-759497475f-tk9q9   1/1     Running            0          31m
maya-grafana-569f594fc7-4jhw5                           0/2     ErrImagePull       0          31m
maya-io-c58c84cb8-nsnz7                                 0/1     Init:2/8           0          31m
maya-ui-bf9c6f9-stpv6                                   1/1     Running            0          31m
memcached-f76c997dd-hmtrg                               1/1     Running            0          31m
mysql-0                                                 2/2     Running            0          31m
od-elasticsearch-logging-0                              1/1     Running            0          31m
od-kibana-logging-6bfcbf8c56-n4gtt                      0/1     ErrImagePull       0          31m
querier-7c665d74fc-b2bcr                                1/1     Running            0          31m
ruler-9d54f9bd9-88pw5                                   0/1     Init:3/6           0          31m
table-manager-6bd648d5f5-bh6x2                          1/1     Running            0          31m

Also for testing, tried to run nginx pod on the cluster. It took ~4mins to pull the image and then the pod came to Running state

  Type    Reason     Age    From                                          Message
  ----    ------     ----   ----                                          -------
  Normal  Scheduled  6m19s  default-scheduler                             Successfully assigned default/ng to director-centos-worker4.mayalabs.io
  Normal  Pulling    6m19s  kubelet, director-centos-worker4.mayalabs.io  Pulling image "nginx"
  Normal  Pulled     2m15s  kubelet, director-centos-worker4.mayalabs.io  Successfully pulled image "nginx"
  Normal  Created    2m15s  kubelet, director-centos-worker4.mayalabs.io  Created container ng
  Normal  Started    2m15s  kubelet, director-centos-worker4.mayalabs.io  Started container ng

[shiveshabhishek]

New status and pod description:

alertmanager-6cb9896d58-hnc78                           0/1     Init:2/3          0          42m
alertstore-6df57df9dc-ct5wk                             0/1     PodInitializing   0          42m
alertstore-tablemanager-6d798c756d-p85ks                1/1     Running           0          42m
cassandra-0                                             1/1     Running           0          42m
chat-server-9fd87877d-kw2r8                             0/1     ErrImagePull      0          42m
cloud-agent-759985db88-2ckg2                            1/1     Running           0          42m
configs-54865d6f6f-r8dfn                                1/1     Running           0          42m
configs-db-74d4fcd6c9-8bxjm                             1/1     Running           0          42m
consul-6c775c669-phf2t                                  1/1     Running           0          42m
distributor-75c86cd857-cqkn5                            1/1     Running           0          42m
elastalert-d8c884dbb-fs87r                              0/1     Init:1/2          0          42m
ingester-5794997bb-lrkj4                                1/1     Running           0          42m
kubera-nginx-ingress-controller-b59jn                   1/1     Running           0          42m
kubera-nginx-ingress-controller-cpk8q                   1/1     Running           0          42m
kubera-nginx-ingress-controller-d6wg4                   1/1     Running           0          42m
kubera-nginx-ingress-controller-hntbg                   1/1     Running           0          42m
kubera-nginx-ingress-controller-t7724                   1/1     Running           0          42m
kubera-nginx-ingress-default-backend-759497475f-rfbp2   1/1     Running           0          42m
kubera-nginx-ingress-default-backend-759497475f-tk9q9   1/1     Running           0          42m
maya-grafana-569f594fc7-4jhw5                           0/2     ErrImagePull      0          42m
maya-io-c58c84cb8-nsnz7                                 0/1     Init:2/8          0          42m
maya-ui-bf9c6f9-stpv6                                   1/1     Running           0          42m
memcached-f76c997dd-hmtrg                               1/1     Running           0          42m
mysql-0                                                 2/2     Running           0          42m
od-elasticsearch-logging-0                              1/1     Running           0          42m
od-kibana-logging-6bfcbf8c56-n4gtt                      0/1     ErrImagePull      0          42m
querier-7c665d74fc-b2bcr                                1/1     Running           0          42m
ruler-9d54f9bd9-88pw5                                   0/1     Init:5/6          0          42m
table-manager-6bd648d5f5-bh6x2                          1/1     Running           0          42m

kubectl describe of the pods:

alertstore-tablemanager-6d798c756d-p85ks

Events:
  Type     Reason     Age                 From                                          Message
  ----     ------     ----                ----                                          -------
  Normal   Scheduled  32m                 default-scheduler                             Successfully assigned kubera/alertstore-tablemanager-6d798c756d-p85ks to director-centos-worker2.mayalabs.io
  Normal   Pulling    32m                 kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal   Pulled     24m                 kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal   Created    24m                 kubelet, director-centos-worker2.mayalabs.io  Created container astm-wait-for-crb
  Normal   Started    24m                 kubelet, director-centos-worker2.mayalabs.io  Started container astm-wait-for-crb
  Normal   Pulling    24m                 kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal   Pulled     15m                 kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal   Created    15m                 kubelet, director-centos-worker2.mayalabs.io  Created container astm-wait-for-cassandra
  Normal   Started    15m                 kubelet, director-centos-worker2.mayalabs.io  Started container astm-wait-for-cassandra
  Warning  Failed     9m22s               kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/alertstore-tablemanager:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/alertstore-tablemanager:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:37586->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     9m22s               kubelet, director-centos-worker2.mayalabs.io  Error: ErrImagePull
  Normal   BackOff    9m21s               kubelet, director-centos-worker2.mayalabs.io  Back-off pulling image "registry.mayadata.io/alertstore-tablemanager:1.11.0-RC4.ee"
  Warning  Failed     9m21s               kubelet, director-centos-worker2.mayalabs.io  Error: ImagePullBackOff
  Normal   Pulling    9m7s (x2 over 15m)  kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/alertstore-tablemanager:1.11.0-RC4.ee"

alertstore

  Type    Reason     Age   From                                          Message
  ----    ------     ----  ----                                          -------
  Normal  Pulling    33m   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Scheduled  33m   default-scheduler                             Successfully assigned kubera/alertstore-6df57df9dc-ct5wk to director-centos-worker2.mayalabs.io
  Normal  Pulled     25m   kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    25m   kubelet, director-centos-worker2.mayalabs.io  Created container init-agent
  Normal  Started    25m   kubelet, director-centos-worker2.mayalabs.io  Started container init-agent
  Normal  Pulling    25m   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Started    16m   kubelet, director-centos-worker2.mayalabs.io  Started container as-wait-for-cassandra
  Normal  Pulled     16m   kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    16m   kubelet, director-centos-worker2.mayalabs.io  Created container as-wait-for-cassandra
  Normal  Pulling    16m   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     11m   kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    11m   kubelet, director-centos-worker2.mayalabs.io  Created container as-wait-for-as-tablemanager
  Normal  Started    11m   kubelet, director-centos-worker2.mayalabs.io  Started container as-wait-for-as-tablemanager
  Normal  Pulling    88s   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/alertstore:1.11.0-RC4.ee"

chat-server-9fd87877d-kw2r8

  Type     Reason     Age                  From                                          Message
  ----     ------     ----                 ----                                          -------
  Normal   Scheduled  34m                  default-scheduler                             Successfully assigned kubera/chat-server-9fd87877d-kw2r8 to director-centos-worker2.mayalabs.io
  Warning  Failed     33m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": failed to copy: http2: server sent GOAWAY and closed the connection; LastStreamID=31, ErrCode=NO_ERROR, debug=""
  Normal   BackOff    33m                  kubelet, director-centos-worker2.mayalabs.io  Back-off pulling image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee"
  Warning  Failed     33m                  kubelet, director-centos-worker2.mayalabs.io  Error: ImagePullBackOff
  Warning  Failed     23m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:34542->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     12m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:36364->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     2m39s (x4 over 33m)  kubelet, director-centos-worker2.mayalabs.io  Error: ErrImagePull
  Warning  Failed     2m39s                kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:39112->35.193.106.198:443: read: connection reset by peer
  Normal   Pulling    2m27s (x5 over 34m)  kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-chatops:1.11.0-RC4.ee"

elast-alert

  Type    Reason     Age   From                                          Message
  ----    ------     ----  ----                                          -------
  Normal  Scheduled  36m   default-scheduler                             Successfully assigned kubera/elastalert-d8c884dbb-fs87r to director-centos-worker2.mayalabs.io
  Normal  Pulling    35m   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     27m   kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    27m   kubelet, director-centos-worker2.mayalabs.io  Created container rl-wait-for-crb
  Normal  Started    27m   kubelet, director-centos-worker2.mayalabs.io  Started container rl-wait-for-crb
  Normal  Pulling    27m   kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     19m   kubelet, director-centos-worker2.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    19m   kubelet, director-centos-worker2.mayalabs.io  Created container am-wait-for-maya-io
  Normal  Started    19m   kubelet, director-centos-worker2.mayalabs.io  Started container am-wait-for-maya-io

maya-grafana

  Type     Reason     Age                  From                                          Message
  ----     ------     ----                 ----                                          -------
  Normal   Scheduled  36m                  default-scheduler                             Successfully assigned kubera/maya-grafana-569f594fc7-4jhw5 to director-centos-worker4.mayalabs.io
  Warning  Failed     30m                  kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.15:56682->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     24m                  kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.15:57318->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     19m                  kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.15:58080->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     14m                  kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": failed to copy: http2: server sent GOAWAY and closed the connection; LastStreamID=31, ErrCode=NO_ERROR, debug=""
  Warning  Failed     13m                  kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": failed to copy: http2: server sent GOAWAY and closed the connection; LastStreamID=23, ErrCode=NO_ERROR, debug=""
  Warning  Failed     9m44s                kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.15:59420->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     9m44s (x3 over 24m)  kubelet, director-centos-worker4.mayalabs.io  Error: ErrImagePull
  Normal   Pulling    9m43s (x4 over 36m)  kubelet, director-centos-worker4.mayalabs.io  Pulling image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee"
  Warning  Failed     4m35s (x4 over 30m)  kubelet, director-centos-worker4.mayalabs.io  Error: ErrImagePull
  Normal   Pulling    4m35s (x4 over 30m)  kubelet, director-centos-worker4.mayalabs.io  Pulling image "registry.mayadata.io/maya-grafana:1.11.0-RC4.ee"
  Warning  Failed     4m35s                kubelet, director-centos-worker4.mayalabs.io  Failed to pull image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/graph-reporter:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.15:59926->35.193.106.198:443: read: connection reset by peer

od-kibana-logging

  Type     Reason     Age                  From                                          Message
  ----     ------     ----                 ----                                          -------
  Normal   Scheduled  39m                  default-scheduler                             Successfully assigned kubera/od-kibana-logging-6bfcbf8c56-n4gtt to director-centos-worker2.mayalabs.io
  Warning  Failed     32m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:33788->35.193.106.198:443: read: connection reset by peer
  Normal   BackOff    32m                  kubelet, director-centos-worker2.mayalabs.io  Back-off pulling image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee"
  Warning  Failed     32m                  kubelet, director-centos-worker2.mayalabs.io  Error: ImagePullBackOff
  Warning  Failed     22m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:35438->35.193.106.198:443: read: connection reset by peer
  Warning  Failed     12m                  kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": failed to copy: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
  Warning  Failed     2m15s (x4 over 32m)  kubelet, director-centos-worker2.mayalabs.io  Error: ErrImagePull
  Warning  Failed     2m15s                kubelet, director-centos-worker2.mayalabs.io  Failed to pull image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee": failed to copy: read tcp 10.43.10.13:40616->35.193.106.198:443: read: connection reset by peer
  Normal   Pulling    2m (x5 over 39m)     kubelet, director-centos-worker2.mayalabs.io  Pulling image "registry.mayadata.io/maya-kibana:1.11.0-RC4.ee"

ruler

  Type    Reason     Age    From                                          Message
  ----    ------     ----   ----                                          -------
  Normal  Scheduled  41m    default-scheduler                             Successfully assigned kubera/ruler-9d54f9bd9-88pw5 to director-centos-worker5.mayalabs.io
  Normal  Pulling    40m    kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     38m    kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    38m    kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-crb
  Normal  Started    38m    kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-crb
  Normal  Pulling    38m    kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     38m    kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    38m    kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-cassandra
  Normal  Started    38m    kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-cassandra
  Normal  Pulling    33m    kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     33m    kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    33m    kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-consul
  Normal  Started    33m    kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-consul
  Normal  Pulling    33m    kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     33m    kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    33m    kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-ingester
  Normal  Started    33m    kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-ingester
  Normal  Pulling    7m35s  kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     7m28s  kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    7m28s  kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-querier
  Normal  Started    7m28s  kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-querier
  Normal  Pulling    7m26s  kubelet, director-centos-worker5.mayalabs.io  Pulling image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Pulled     7m25s  kubelet, director-centos-worker5.mayalabs.io  Successfully pulled image "registry.mayadata.io/maya-init:1.0.0"
  Normal  Created    7m25s  kubelet, director-centos-worker5.mayalabs.io  Created container rl-wait-for-alertmanager
  Normal  Started    7m24s  kubelet, director-centos-worker5.mayalabs.io  Started container rl-wait-for-alertmanager