openZITI in Kubernetes Cluster

1

Hello,

we are trying very hard for days now to get openziti controller running in a kubernetes cluster. We tried several times, different setups like quick start guide with minikube or the normal documentation and a minimal controller setup running on a cluster deployed by kubekey with no success. the problem is always the same, have a look at the failing pod description:

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    2m15s                 default-scheduler  Successfully assigned ziti-controller/ziti-controller-55fcf485bf-kp9hn to miniziti
  Warning  FailedMount  119s (x6 over 2m14s)  kubelet            MountVolume.SetUp failed for volume "cert-web-identity" : secret "ziti-controller-web-identity-secret" not found
  Warning  FailedMount  119s (x6 over 2m14s)  kubelet            MountVolume.SetUp failed for volume "cert-ctrl-plane-identity" : secret "ziti-controller-ctrl-plane-identity-secret" not found
  Warning  FailedMount  119s (x6 over 2m14s)  kubelet            MountVolume.SetUp failed for volume "cert-edge-signer" : secret "ziti-controller-edge-signer-secret" not found
  Warning  FailedMount  103s (x7 over 2m14s)  kubelet            MountVolume.SetUp failed for volume "ziti-controller-ctrl-plane-cas" : configmap "ziti-controller-ctrl-plane-cas" not found

I feel like the documentation is out of date. possible? regards

1 Like
2

Welcome to the forum, @fkaute.

I suspect you're facing the same issues as another user did recently.

Let's check a couple of things.

Ensure that Trust Manager's "trust-namespace" value is set to the same namespace where the Ziti Controller is installed, because that's where the certificates are located from which Trust Manager will compose the Bundle resource.

The error message you shared points to the Bundle that provides CM ziti-controller-ctrl-plane-cas not being available, which would normally be pending until Trust Manager is able to source the necessary certs.

You can diagnose this by describing the Trust Manager controller and following the TM log.

Another thing to check is that Helm's wait option is false, which is the default for the CLI, but some wrappers like the Terraform Provider set it to true by default, which prevents the life cycle hooks from triggering.

3

first of all I would like to mention that in this quickstart guide Kubernetes Quickstart | OpenZiti
certmanager and trustmanager are only mentioned to be automatically created when installing the controller. that does not work, see error above. No certmanager or trustmanager running at all when following the k8s quickstart guide

Now I luckily began with this doc: Install OpenZiti Controller in Kubernetes | OpenZiti
Here is what I do to install cert/trustmanager and controller:

Install cert manager by kubectl apply as mentioned here kubectl apply - cert-manager Documentation

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

Next, I install trustmanager as described here trust-manager - cert-manager Documentation

helm repo add jetstack https://charts.jetstack.io --force-update
#helm upgrade -i -n cert-manager cert-manager jetstack/cert-manager --set installCRDs=true --wait --create-namespace
helm upgrade -i -n cert-manager trust-manager jetstack/trust-manager --wait

I commented out the second line that installs cert-manager because I already did this before. Next I add the openziti charts repo

helm repo add openziti https://docs.openziti.io/helm-charts/

install the CRDs

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml
kubectl apply -f https://raw.githubusercontent.com/cert-manager/trust-manager/v0.4.0/deploy/crds/trust.cert-manager.io_bundles.yaml

and install the controller with helm as mentioned:

helm install \
    --namespace ziti-controller ziti-controller-minimal1 \
    openziti/ziti-controller \
        --set clientApi.advertisedHost="ziti-controller-minimal.example.com" \
        --set clientApi.advertisedPort="443"

resulting into the same problem mentioned in OP. I have a little clue now what is the problem with the trust-namespace value but

a) how am I setting this value to my controller namespace with 'helm upgrade -i -n cert-manager trust-manager jetstack/trust-manager --wait'?
b) I feel like Im supposed to create an own helm chart for trust manager / controller but is there an example on how to do it?

EDIT: chatGPT helped out here:

helm upgrade -i -n cert-manager trust-manager jetstack/trust-manager --wait --set app.trust.namespace=openziti

Now the controller is up and running. Now I logged into the controller pod, logged in and created a router with a jwt token as described here Install OpenZiti Router in Kubernetes | OpenZiti

Lastley, when I try to install the router with helm as described, the pod does not get created due to:

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    47s                default-scheduler  Successfully assigned openziti/ziti-router-1701773138-7cdf56f697-qvmgw to miniziti
  Warning  FailedMount  16s (x7 over 48s)  kubelet            MountVolume.SetUp failed for volume "ziti-router-identity" : secret "ziti-router-1701773138-identity" not found

even though the router gets installed into the same namespace as the controller and the same namespace as used as app-namespace for the trust-manager.

But I just saw thats exactly the same error now as mentioned by @qrkourier in the other issue topic so I try to catch up on this one for now

1 Like
4

meanwhile I managed to get a controller and a router online but facing new problems, will open another post and link it here, thank you

1 Like
5

Nice job working through that issue with Trust Manager's trust-namespace parameter.

You're correct, there's no requirement to use the Trust Manager or Cert Manager sub-charts that are defined in the OpenZiti Controller chart, though it may be convenient when their custom resources are not already defined in the cluster, e.g., Bundle, Issuer, Certificate, etc.

The local Kubernetes (minikube) quickstart uses this approach to set the value of trust-manager.app.trust.namespace to the same namespace where the OpenZiti Controller is installed. It does this after defining the necessary custom resources like you did with kubectl apply.

    helmWrapper upgrade --install "ziti-controller" "${ZITI_CHARTS_REF}/ziti-controller" \
        --namespace "${ZITI_NAMESPACE}" --create-namespace \
        --set clientApi.advertisedHost="miniziti-controller.${MINIZITI_INGRESS_ZONE}" \
        --set trust-manager.app.trust.namespace="${ZITI_NAMESPACE}" \
        --set trust-manager.enabled=true \
        --set cert-manager.enabled=true \
        --values "${ZITI_CHARTS_URL}/ziti-controller/values-ingress-nginx.yaml" >&3

link to local Kubernetes quickstart BASH script: ziti/quickstart/kubernetes/miniziti.bash at main · openziti/ziti · GitHub

Besides scripting Helm commands I've also worked with Terraform modules a bit. Here's an example of a root module using the OpenZiti Controller module.