Issues with minimal controller deployment

[icsy7867]

I am having a hard time getting the controller deployment working via helm chart.

I am currently just trying the simple deployment:
https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/README.md

The first thing i notice, is that when I use:
--set trust-manager.enabled="true"

I get a YAML/JSON error:

Error: INSTALLATION FAILED: YAML parse error on ziti-controller/charts/trust-manager/templates/role.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Release.Namespace":interface {}(nil)}

I also copied the default values.yaml from github, and tried editing directly there. It seems there is an issue when trying to pull the subchart. To get around this, I just set this to false and manually installed:
https://cert-manager.io/docs/trust/trust-manager/installation/

But the original errors were still persisting.

I am also seeing the ziti-controller init container having issues starting...

The PVC apepars to be bound, and the nodes are ready. So I am not sure what is going on here. Any thoughts or ideas? Thanks!

• EDIT *
  So, this line in the default YAML file:

namespace: "{{ .Release.Namespace }}"

Does not appear to parse correctly.

[qrkourier]

Are you deploying to your cluster directly in one step with a helm upgrade --install command or Helm wrapper, or are you deploying in multiple steps like rendering a template to Kustomize, GitOps, etc.?

The controller needs CM and TM, and you can either install them as sub-charts in the same namespace or your can use an existing instance of either or both. TM's "trust-namespace" must be set to where ziti-controller is installed so it can build Bundle resources from Ziti's root CAs.

[qrkourier]

If using a pre-installed, existing instance of TM, then install ziti-controller with --set trust-manager.enabled="false"

[icsy7867]

Are you deploying to your cluster directly in one step with a helm upgrade --install command or Helm wrapper, or are you deploying in multiple steps like rendering a template to Kustomize, GitOps, etc.?

The controller needs CM and TM, and you can either install them as sub-charts in the same namespace or your can use an existing instance of either or both. TM's "trust-namespace" must be set to where ziti-controller is installed so it can build Bundle resources from Ziti's root CAs.

I just used the minimal install example as found here:
https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/README.md#minimal-installation

helm install ...

[icsy7867]

Alright, part of the problem seemed to be using a really old CRD YAML file. The website listed 0.7.0, but i used:
kubectl apply -f https://raw.githubusercontent.com/cert-manager/trust-manager/v0.16.0/deploy/crds/trust.cert-manager.io_bundles.yaml

And things seemed to deploy a bit smoother. Now I am struggling with getting the router to be able to talk with the controller. Kubernetes is resolving the service addresses for the various ports, but the communication isn't working. However if I use the pods IP address instead of the service/ClusterIP address it seems to work (I can hit https://pod-ip-address:1280 and get the proper response.

Almost there... I wiped and redeployed, and the services appear to work. I can hit:
https://ziti-controller-ctrl:443 and get the proper JSON back. However the router is still erroring out:

{"cause":"token is unverifiable: error while executing keyfunc: could not retrieve token URL certificate: could not contact remote server [https://ziti-controller:1280]: Get \"https://ziti-controller:1280\": dial tcp 10.41.11.131:1280: connect: connection refused","file":"github.com/openziti/ziti/router/enroll/enroll.go:98","func":"github.com/openziti/ziti/router/enroll.(*RestEnroller).Enroll","level":"fatal","msg":"failed to parse JWT","time":"2025-02-28T16:04:38.062Z"}

And this is a bit odd, no matter what I set ctrl.endpoint to, the router log indicates that it is ALWAYS try to hit:
https://ziti-controller:1280, which does not work.

So I got around this... I must a service for 1280 -> 1280 named 'ziti-controller'. This seemed to work fine. The router appears to be happy and talking. Though I cannot figure out how to tell the router to talk to anything other thank ziti-controller:1280

[qrkourier]

Good progress!

The Trust Manager CRDs

The version was pinned to not-even-close-to-latest because TM introduced a schema change that would have forced a breaking change in the controller chart for users who opt to install TM as a sub-chart. I reasoned the older CRD is "probably fine" since Ziti's use case for TM is straightforward and unlikely to evolve substantially.

As an aside, The only utility of TM for Ziti is distributing the root CAs as a Bundle resource throughout the cluster. I'm contemplating replacing that with the ClusterTrustBundle resource that's being introduced in K8S, when development stabilizes. It would be nice to prune a dependency.

Router to Control Plane Connectivity

You're correct it should not be necessary to configure the controller to listen at ziti-controller:1280. The router should obey the ctrl.endpoint input after enrollment. The enrollment operation, which occurs on the first run, will extract the controller's URL from the enrollment token (JWT).

Do I understand correctly? You installed the router chart with the input value ctrl.endpoint=ziti-controller-ctrl:1280 (the unqualified domain name and port of the controller's router control plane (ctrl) ClusterIP Service in the same namespace as the router), and the rolled-out pod is ignoring that and stubbornly connecting to ziti-controller:1280 instead?

If that's an accurate summary, the first thing I'd confirm is the router pod which is attempting to connect to the wrong address was rolled-out with the updated pod spec, following a Helm upgrade operation. Updating the template in the router deployment by performing a helm upgrade should have triggered an immediate rollout, so let's check one of the newly-rolled-out router pods' control plane endpoint address.

Here's my local test env's router configuration in the ConfigMap declared by the chart.

kubectl get configmap -n miniziti ziti-router-config -o yaml \
|  yq '.data."ziti-router.yaml" | from_yaml | .ctrl.endpoint'

Confirming the rolled-out pod's mounted ConfigMap has the same value.

kubectl -n miniziti get pods --selector app.kubernetes.io/name=ziti-router -o jsonpath="{.items[0].metadata.name}" \
| xargs -rI POD kubectl exec -n miniziti POD -- cat /etc/ziti/config/ziti-router.yaml \
| yq '.ctrl.endpoint'

[icsy7867]

Those values do match...

AND... after checking the container it is doing exactly what it is supposed to. It appears to be hitting the correct endpoint now. I tried redploying many many times and was still getting the same error in the container log... so sorry about that. It seems happy now!

{"endpoint":"tls:ziti-controller-client:443"," ...

I'm so close. The router and controller appear to be happy...

I am now. trying to deploy zrok... But getting "unauthenticated"

panic: error connecting to the ziti edge management api: [POST /authenticate][401] authenticateUnauthorized  &{Error:0xc0002b2600 Meta:0xc0003e2080}

I am following:
https://github.com/openziti/helm-charts/blob/main/charts/zrok/README.md

And modified a bit...

ZITI_PWD=$(kubectl -n "${ZITI_NAMESPACE}" get secrets "ziti-controller-admin-secret" \
    --output go-template='{{index .data "admin-password" | base64decode}}')

This does give out a password. Through various troubleshooting steps, I am trying to put this in manually.

helm upgrade --install \
    --namespace "zrok-test" zrok \
    --values https://openziti.io/helm-charts/charts/zrok/values-ingress-nginx.yaml \
    --set "ziti.advertisedHost=ziti-controller-mgmt" \
    --set "ziti.advertisedPort=443" \
    --set "ziti.password=PASSWORD" \
    --set "dnsZone=${ZROK_DNS_ZONE}" \
    --set "controller.ingress.hosts[0]=ctrl.${ZROK_DNS_ZONE}" \
    --set "persistence.existingClaim=zrok-data" \
    --set "persistence.accessMode=ReadWriteMany" \
    --set "controller.persistence.accessMode=ReadWriteMany" \
    openziti/zrok

I noticed the init container was trying to access:
https://ziti-controller-mgmt:443/.well-known/est/cacerts

And I can confirm that this does return a CACERT. So I know that service name, and port are value and valid:
curl -k https://ziti-controller-mgmt:443/.well-known/est/cacerts works in another container.

This is sort of interesting... So if I shell into the ziti controller container and run:

ziti edge login

if I use port 1280:
"message": "path /edge/management/v1/authenticate was not found"

but using 1281, the password generated as a secret does not work:

        "code": "INVALID_AUTH",
        "message": "The authentication request failed",
        "requestId": "8Fqr0Abem"

[icsy7867]

Woo! So I figured it was user error again, and I Just wiped away my entire namespace and started over. I verified I could do a ziti edge login prior to proceeding...

zrok is running!

[qrkourier]

Yay! Nice work. If you have any clarifications or corrections for the README.md.gotmpl files, I'd welcome a pull request.

Looking forward: zrok is getting ready to drop v1, currently v0.4. There will be an upgrade path for your K8S deployment, and it may or may not involve a chart major revision, depending on whether it proves necessary to introduce a breaking change to the chart.

TL;DR You might want to "pin" the chart version and make sure you have database backups in preparation for the upgrade, i.e., to make sure you have a chance to opt-in to an irreversible database migration.

[nguu0123]

should we find a way to fix this in the chart, specifying the values.yml like this work for me:

clientApi:
  advertisedHost: "ziti-controller-minimal.example.com"
  advertisedPort: 443
cert-manager:
  enabled: true
trust-manager:
  enabled: true
  app:
    trust:
      # Match this namespace where you install the chart
      namespace: "ziti-controller"

[qrkourier]

No, we'll de-couple CM/TM from the Ziti controller, so CM/TM will stop being "subcharts" and dependencies of the Ziti controller chart. Instead, it will be a prerequisite to deploy CM/TM and configure TM's "trust namespace" to the namespace where Ziti controller is deployed.