APIForbiddenError & APIServerError handling

[himani-dangwal]

Keywords

APIServerError APIForbiddenError

Problem

I am new to kopf . . when My pod is spun and kopf is triggered I need to wait around 20-30 sec and post that it logs below error .
moreover, the scope of kopf is only namespace based
Request attempt #1/9 failed; will retry: GET https://10.xxx.x.x:443/apis/custom.metrics.k8s.io/v1beta1 -> APIServerError(None, None)

Request attempt #7/9 failed; will retry: GET https://10.100.0.1:443/api/v1/namespaces -> APIForbiddenError('namespaces is forbidden: User "system:serviceaccount:xxx:xxx" cannot list resource "namespaces" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'namespaces is forbidden: User "system:serviceaccount:xxx:xxx" cannot list resource "namespaces" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'kind': 'namespaces'}, 'code': 403})

is there any config setting that I am missing

[amannocci]

Hi @himani-dangwal, you need to follow the RBAC documentation to allow kopf to watch resources.
Ref: https://kopf.readthedocs.io/en/stable/deployment/#rbac

[PidgeyBE]

I'm facing the same issue. The docs mention:

Here is an example of what an RBAC config should look like (remove the parts which are not needed: e.g. the cluster roles/bindings for the strictly namespace-bound operator):

But without the clusterroles, kopf is having the issue raised here and it takes 30s to start up.

[amannocci]

Can you give an example of the manifest you are using ?

[f-helm]

@amannocci I have the same issue. Easiest way to reproduce should be to start kopf with the argument --namespace=my-app.

My command to start kopf is this:
kopf run /src/kopf_operator.py --namespace=my-app --verbose

This log line indicates that something is wrong (note keyword "cluster-wide"):

kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.

After this log line, i get the same errors already reported.

[DrummyFloyd]

i've got similar issue when i deploy kopf as namespace-scope action

kopf run --namespace pg-default
kopf run --namespace default-pg

with following role/rolebinding

# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: "crui-crunchy-userinit-controller-role"
  namespace: "default-pg"
rules:
  # Framework: posting events about handler progress/errors (optional but useful for debugging)
  - apiGroups: [""]
    resources: [events]
    verbs: [create]
# Needed to read secreets on specified namespace
  - apiGroups: [""]
    resources: [secrets]
    verbs: [get, list, watch, update, patch]
---
# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: "default-pg"
  name: "crui-crunchy-userinit-controller"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: "crui-crunchy-userinit-controller-role"
subjects:
  - kind: ServiceAccount
    name: "crui-crunchy-userinit-controller"
   namespace: "default-pg"
---

got this error

Request attempt #8/9 failed; will retry: GET https://kubernetes.default.svc/api/v1/namespaces -> APIForbiddenError('namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'kind': 'namespaces'}, 'code': 403})

[amannocci]

i've got similar issue when i deploy kopf as namespace-scope action

kopf run --namespace pg-default

with following role/rolebinding

# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: "crui-crunchy-userinit-controller-role"
  namespace: "default-pg"
rules:
  # Framework: posting events about handler progress/errors (optional but useful for debugging)
  - apiGroups: [""]
    resources: [events]
    verbs: [create]
# Needed to read secreets on specified namespace
  - apiGroups: [""]
    resources: [secrets]
    verbs: [get, list, watch, update, patch]
---
# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: "default-pg"
  name: "crui-crunchy-userinit-controller"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: "crui-crunchy-userinit-controller-role"
subjects:
  - kind: ServiceAccount
    name: "crui-crunchy-userinit-controller"
   namespace: "default-pg"
---

got this error

Request attempt #8/9 failed; will retry: GET https://kubernetes.default.svc/api/v1/namespaces -> APIForbiddenError('namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'kind': 'namespaces'}, 'code': 403})

Did you deploy also the ClusterRole with needed cluster wide permissions?
In the above example, the namespace used with the CLI pg-default doesn't match the namespace set in the RoleBinding resource default-pg.

[DrummyFloyd]

i've got similar issue when i deploy kopf as namespace-scope action
kopf run --namespace pg-default
with following role/rolebinding

# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: "crui-crunchy-userinit-controller-role"
  namespace: "default-pg"
rules:
  # Framework: posting events about handler progress/errors (optional but useful for debugging)
  - apiGroups: [""]
    resources: [events]
    verbs: [create]
# Needed to read secreets on specified namespace
  - apiGroups: [""]
    resources: [secrets]
    verbs: [get, list, watch, update, patch]
---
# Source: crunchy-userinit-controller/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: "default-pg"
  name: "crui-crunchy-userinit-controller"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: "crui-crunchy-userinit-controller-role"
subjects:
  - kind: ServiceAccount
    name: "crui-crunchy-userinit-controller"
   namespace: "default-pg"
---

got this error

Request attempt #8/9 failed; will retry: GET https://kubernetes.default.svc/api/v1/namespaces -> APIForbiddenError('namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'namespaces is forbidden: User "system:serviceaccount:default-pg:crui-crunchy-userinit-controller" cannot list resource "namespaces" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'kind': 'namespaces'}, 'code': 403})

Did you deploy also the ClusterRole with needed cluster wide permissions? In the above example, the namespace used with the CLI pg-default doesn't match the namespace set in the RoleBinding resource default-pg.

the namespace used with the CLI pg-default doesn't match the namespace set in the RoleBinding resource default-pg.

typo on my comment , the correct command is kopf run --namespace default-pg

Did you deploy also the ClusterRole with needed cluster wide permissions?

No because i expect to have a controller namespace scope that why

[DrummyFloyd]

any update ?