Im trying to get a working k8s openziti cluster running. I mainly followed this k8s quickstart guide here Kubernetes Quickstart | OpenZiti
but before that I installed cert-manager and trust-manager like this:
#installed cert-manager with k apply:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
#installed trust-manager with helm and the controller namespace as app-namespace:
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade -i -n cert-manager trust-manager jetstack/trust-manager --wait --set app.trust.namespace=ziti-controller
Now I went for every step in the k8s quickstart guide except installing the tunneler app. When I try to install the router it fails because it cant find the controllers host, see pod logs:
ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 0.065] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 0.127] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 0.281] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 0.503] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 0.839] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 1.172] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)] endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262]} unable to connect controller
[ 1.749] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)] endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262]} unable to connect controller
[ 2.934] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)] endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262]} unable to connect controller
[ 4.211] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)] endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262]} unable to connect controller
[ 6.188] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 10.481] ERROR ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:ziti-controller-ctrl.miniziti.svc:6262] error=[error connecting ctrl (dial tcp: lookup ziti-controller-ctrl.miniziti.svc on 10.96.0.10:53: no such host)]} unable to connect controller
[ 15.007] FATAL ziti/router.(*Router).startControlPlane.func1: unable to connect to any controllers before timeout
now I did the mentioned edit for the names in /etc/hosts and I edited the coredns config of minikube. I also deleted the pod. here is the dnstest output:
Let's check the cluster namespaces. It looks like you're creating the OpenZiti Controller's Helm release in namespace miniziti, which is the default namespace used by miniziti.bash, but the Trust Manager release has input value app.trust.namespace=ziti-controller, which should be app.trust.namespace=miniziti so that Trust Manager can compose trust Bundle resources from the Certificate resources in the Controller's namespace.
I see the error from the OpenZiti Router's pod log too. I'm assuming this means the Router's Helm release has input value ctrl.endpoint=ziti-controller-ctrl.miniziti.svc:443. I think it's likely that Kubernetes service is not yet available if the OpenZiti Controller pod is still waiting for Trust Manager, so this problem may resolve itself within a few seconds of the OpenZiti Controller becoming ready.
What's the status of the Controller's pod?
kubectl get pods --selector app.kubernetes.io/component=ziti-controller
In summary, it looks like the Router is waiting for the Controller, and the Controller is waiting for Trust Manager to provide the trust Bundle resource.
Hi @qrkourier
not sure why you assume the namespace must be miniziti since the quick start documentation mentioned above is using the ziti-controller ns when installing controller with helm:
Thanks for confirming your Controller and Trust Manager are ready. This leads me to believe the Helm input value ctrl.endpoint to the Router's Helm release is not valid for the "ctrl" (Router control plane) service provided by the OpenZiti Controller.
The error logs indicate this value must be ziti-controller-ctrl.miniziti.svc:6262, and this must be changed to match the Kubernetes service provided by the OpenZiti Controller pod named like "ziti-controller-ctrl."
For example, if your Controller is deployed in namespace "ziti-controller," and the Controller's Helm release has input value ctrlPlane.advertisedPort=443 (the default). A valid value for the Router's release is ctrl.endpoint=ziti-controller-ctrl.ziti-controller.svc:443.
You can upgrade your Router release with this value by re-running helm upgrade with a complete set of input values. You can extract the current values with helm get values.
I see what you mean about the incongruity between the manual steps that use "ziti-controller" as an example and the scripted alternative that uses namespace "miniziti" as a default.
I will correct those manual steps to use miniziti.bash default values as an example. That way, there won't be a discrepancy between the namespaces if you follow the manual steps or run the script.
Sorry that made you stumble! Will you let me know if correcting the value of ctrl.endpoint gets your Router up and running?
