Hello All,
I am trying to self-host zrok in a k8s cluster. I have tried to deploy a ziti-controller first by following: helm-charts/charts/ziti-controller/README.md at main · openziti/helm-charts · GitHub, and my ziti-controller pod is stuck at Init status with Warning like:
Warning FailedMount 2m5s (x4 over 2m8s) kubelet MountVolume.SetUp failed for volume "cert-ctrl-plane-identity" : secret "ziti-controller-minimal-ctrl-plane-identity-secret" not found
Warning FailedMount 2m5s (x4 over 2m8s) kubelet MountVolume.SetUp failed for volume "cert-web-client-identity" : secret "ziti-controller-minimal-web-client-identity-secret" not found
Warning FailedMount 2m5s (x4 over 2m8s) kubelet MountVolume.SetUp failed for volume "cert-edge-signer" : secret "ziti-controller-minimal-edge-signer-secret" not found
Warning FailedMount 2m5s (x4 over 2m8s) kubelet MountVolume.SetUp failed for volume "ziti-controller-ctrl-plane-cas" : configmap "ziti-controller-minimal-ctrl-plane-cas" not found
Warning FailedMount 2m5s (x4 over 2m8s) kubelet MountVolume.SetUp failed for volume "cert-web-identity" : secret "ziti-controller-minimal-web-identity-secret" not found
Warning FailedMount 2m1s (x5 over 2m8s) kubelet MountVolume.SetUp failed for volume "cert-ctrl-plane-client-identity" : secret "ziti-controller-minimal-ctrl-plane-client-identity-secret" not found
Not sure what to do. Please help.
Hello and welcome to the community forum, @stanworld.
That's the correct first step. I can help you troubleshoot the Ziti controller deployment in your K8S cluster.
After that, the next steps will be deploy a router and then deploy zrok.
The events you pasted are typical during the first startup because of the async nature of k8s. If these do not resolve after a few minutes there must be another problem.
There will be an error message in one of the pod logs that are created by your deployment.
Perhaps it will be easier to verify your config if you provide sanitized Helm input values. Which K8S distro are you using? How will you be publishing services, with a load balancer, ingress controller, etc.?
Hello @qrkourier , thanks for the reply! I have a single node cluster k8s 1.30 set up using kubeadm. I am thinking using a ingress controller like nginx but I have not deployed any. There is no other things deployed in the cluster.
All I did is based on helm-charts/charts/ziti-controller/README.md at main · openziti/helm-charts · GitHub,
- helm repo add openziti OpenZiti Helm Charts | helm-charts
- kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml
kubectl apply -f https://raw.githubusercontent.com/cert-manager/trust-manager/v0.9.0/deploy/crds/trust.cert-manager.io_bundles.yaml
- helm install
--namespace ziti-controller ziti-controller-minimal1
openziti/ziti-controller
--set clientApi.advertisedHost="ziti-controller-minimal.example.com"
--set clientApi.advertisedPort="443"
Then the ziti-controller pod is stuck with the description like the above. I checked kubectl get secret -A, and there is no secrets created like: ziti-controller-minimal-ctrl-plane-identity-secret,
Not sure what steps are missing. I am quite new to k8s and ziti at the same time, so I am really stuck.
1 Like
Thanks for sharing those specifics.
Troubleshooting the Controller Deployment
The symptom is that the controller deployment events reveal that some required volumes are unavailable. The controller requires volumes provided by Cert Manager and Trust Manager, so I'm guessing one or both of those deployments do not exist or their logs show an error.
The minimal install example assumes you have installed Cert Manager and Trust Manager. I should clarify that in the README (link to updated doc preview). You can install them as sub-charts in the same k8s namespace by adding these input values.
helm upgrade --install \
--namespace ziti-controller ziti-controller-minimal1 \
openziti/ziti-controller \
--set clientApi.advertisedHost="ziti-controller-minimal.example.com" \
--set clientApi.advertisedPort="443" \
--set cert-manager.enabled="true" \
--set trust-manager.enabled="true"
Publishing Ziti's TLS Servers on the Cluster Edge
Ziti controller and router deployments that terminate TLS at the pod and must read the TLS ClientHello to function, e.g., mTLS, SNI cert selection, etc. While some older Ingress controllers have such annotations to enable TLS passthrough (an L4 TCP proxy), Traefik calls this a TCPRoute, Gateway API calls it a TLSRoute.
I can confirm that ingress-nginx works when configured for what inress-nginx calls "ssl-passthrough." It's a startup flag you can patch into the deployment. You must annotate your ingressClassName: nginx Ingress resources for passthrough as well. There are examples in the README and a separate file values-ingress-nginx.yaml.
Hi @qrkourier ,
I've tried:
helm upgrade "ziti-controller" openziti/ziti-controller
--namespace miniziti --create-namespace --install
--set clientApi.advertisedHost="miniziti-controller.miniziti.internal"
--set trust-manager.enabled=true
--set cert-manager.enabled=true
--set trust-manager.app.trust.namespace="miniziti"
--values https://openziti.io/helm-charts/charts/ziti-controller/values-ingress-nginx.yaml,
However,
and when I describe the controller pod, it suggests
Warning FailedScheduling 29m (x178 over 15h) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
Great! Several of the problems you reported were solved by deploying Cert Manager and Trust Manager. The ziti-controller deployment waits for the cluster to bind the declared PersistentVolumeClaim (PVC).
Many Kubernetes distributions provide a default storage class for PVCs. Please investigate why the cluster did not bind the PVC.
kubectl describe nodes
kubectl describe pvc \
--namespace=miniziti \
--selector=app.kubernetes.io/component=ziti-controller
