Helm v4 Incompatibility, Ziti-Router Unresponsive

Support General Questions
1

I upgraded networking equipment, and both ziti-controller and ziti-router threw a fit (running in kubernetes).

I found that there is a bug/incompatibility with helm version 4.

When installing the helm chart, debugging shows this:

level=DEBUG msg="Error creating resource via patch" namespace=ziti name=ziti-controller gvk="apps/v1, Kind=Deployment" error="failed to create typed patch object (ziti/ziti-controller; apps/v1, Kind=Deployment): .spec.template.spec.containers[name=\"ziti-controller\"].ports: duplicate entries for key [containerPort=1280,protocol=\"TCP\"]" level=WARN msg="upgrade failed" name=ziti-controller error="failed to create resource: failed to create typed patch object (ziti/ziti-controller; apps/v1, Kind=Deployment): .spec.template.spec.containers[name=\"ziti-controller\"].ports: duplicate entries for key [containerPort=1280,protocol=\"TCP\"]" Error: UPGRADE FAILED: failed to create resource: failed to create typed patch object (ziti/ziti-controller; apps/v1, Kind=Deployment): .spec.template.spec.containers[name="ziti-controller"].ports: duplicate entries for key [containerPort=1280,protocol="TCP"]

I downgraded to helm version 3.19.2, which allowed the controller chart to install.

I re-enrolled my controller, and installed the chart with this command:

helm upgrade --install openziti/ziti-router --namespace ziti --create-namespace --debug --version 2.1.0 --values values-router-2.1.0.yaml --set-file enrollmentJwt=/tmp/main-router.jwt

The pod starts, but only shows this in the logs:

INFO: config file exists in /etc/ziti/config/ziti-router.yaml
{"file":"github.com/openziti/ziti/router/env/config_edge.go:398","func":"github.com/openziti/ziti/router/env.(*EdgeConfig).loadCsr","level":"info","msg":"loaded csr info from configuration file at path [edge.csr]","time":"2026-03-19T03:43:05.673Z"}

After a short while, the liveness probe fails, and the pod restarts.

I see nothing in the controller logs from the router trying to connect.

2

Hey @thedarkula! I recognize that error. There was a bug in some versions of the ziti-controller chart that caused the podspec to have an illegal, redundant declaration for the same server port on the container. There can be only one per port.

If you're running the ctrlPlane and clientApi with the same advertisedAddress and advertisedPort (that's been the default for some time, usually setting only clientApi and letting ctrlPlane piggy-back on clientApi's cluster service, ingress, etc.), then you can work around it by disabling the redundant ClusterIP service with input value ctrlPlane.service.enabled=false.

I'd be very interested to know if the latest, stable Helm Chart still has this issue.

Additionally, Ziti 2.0.0 prereleases are available in the Helm repo so you can get a jump on that upgrade step. The main difference with the new controller chart, major version 3 (w/ Ziti v2), is the requirement to specify your intent for controller clustering, e.g., cluster.mode=standalone if you do not wish to migrate the controller datastore to HA cluster mode.

❯ helm repo update openziti
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "openziti" chart repository
Update Complete. ⎈Happy Helming!⎈

❯ helm search repo "openziti/ziti-controller"
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
openziti/ziti-controller        3.1.1           1.7.2           Host an OpenZiti controller in Kubernetes

❯ helm search repo "openziti/ziti-controller" --devel
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
openziti/ziti-controller        3.2.0-pre4      2.0.0-pre5      Host an OpenZiti controller in Kubernetes
3

Wonderful! I am working on the upgrades now.

I did see this note in the new controller values:

# DEPRECATION NOTICE: The separate web-binding root CA is deprecated.
# Web certificates are now issued from the edge-signer-issuer (unified PKI).
# The webBindingPki.alternativeIssuer option is preserved for backward compatibility
# but the web-root-cert and web-root-issuer resources are orphaned.
webBindingPki:

I do have this section from the 2.x.x chart version

webBindingPki:
  # -- generate a separate PKI root of trust for web bindings, i.e., client,
  # management, and prometheus APIs
  enabled: true
  altServerCerts:
    zac:
      # -- request an alternative server certificate from a cert-manager issuer
      mode: certManager
      # -- name of the tls secret for cert-manager to create and manage the server
      #    certificate and private key
      # -- request a certificate for these alternative names distinct from advertisedHost and dnsNames of the clientApi, managementApi, and ctrlPlane
      altDnsNames:
        - zac.domain.com
      secretName: ziti-controller-zac-alt-server-cert
      # -- issuer ref to use when requesting the alternative server certificate
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-production
      # -- where to mount the tls secret on the pod - must not collide with another mountpoint
      mountPath: /etc/ziti/alt-server-cert-zac
    mgmt:
      # -- request an alternative server certificate from a cert-manager issuer
      mode: certManager
      # -- name of the tls secret for cert-manager to create and manage the server
      #    certificate and private key
      # -- request a certificate for these alternative names distinct from advertisedHost and dnsNames of the clientApi, managementApi, and ctrlPlane
      altDnsNames:
        - ziti-mgmt.domain.com
      secretName: ziti-controller-mgmt-alt-server-cert
      # -- issuer ref to use when requesting the alternative server certificate
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-production
      # -- where to mount the tls secret on the pod - must not collide with another mountpoint
      mountPath: /etc/ziti/alt-server-cert-mgmt

What is the recommended way to incorporate this in version 3.x.x?

4

For anyone running into the conflicting ports issue, my fix to keep everything separate is this:

ctrlPlane:
  # -- cluster service target port on the container
  # containerPort: "{{ .Values.clientApi.containerPort }}"
  containerPort: 1283
5

Using the new 3.0.0-pre4 router chart, In the values file, I only changed this:

edge:
  advertisedHost: ziti-router.domain.com

I re-enrolled the router in the controller.

When I install the router chart, the pod logs only show this:

INFO: config file exists in /etc/ziti/config/ziti-router.yaml
{"file":"github.com/openziti/ziti/v2/router/env/config_edge.go:398","func":"github.com/openziti/ziti/v2/router/env.(*EdgeConfig).loadCsr","level":"info","msg":"loaded csr info from configuration file at path [edge.csr]","time":"2026-03-19T17:20:11.928Z"}
{"file":"github.com/openziti/ziti/v2/router/env/config_edge.go:329","func":"github.com/openziti/ziti/v2/router/env.parseEdgeListenerOptions","level":"info","msg":"advertised port [0] in [listeners[443].options.advertise] does not match the listening port [0] in [listeners[3022].address].","time":"2026-03-19T17:20:11.928Z"}

Describing the router pod shows these failures:

  Warning  Unhealthy  6m9s                    kubelet            spec.containers{ziti-router}: Readiness probe failed:
  Warning  Unhealthy  5m19s (x10 over 7m19s)  kubelet            spec.containers{ziti-router}: Liveness probe failed: Error: no processes found matching filter, use 'ziti agent list' to list candidates

I do not see any attempt to connect from the router in the controller logs.

6

The joys of new networking equipment :slight_smile:

I sorted out a tonne of things.
I needed a SNAT rule to allow LAN devices to connect to each other.
It does feel like the ziti-router needs to say that it cannot connect instead of failing silently.

I also found that running ziti edge re-enroll edge-router produces a functional jwt, but using the ZAC and clicking on re-enroll, followed by downloading the jwt does not.
The router says that it is an invalid token if downloaded from the ZAC.

Now, when I bring up the controller, I see this on repeat in the logs:

{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:41668","time":"2026-03-19T20:22:55.358Z"}

Installing the router shows this in the controller logs:

{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34228","time":"2026-03-19T20:35:00.536Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34234","time":"2026-03-19T20:35:02.184Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34248","time":"2026-03-19T20:35:02.263Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34254","time":"2026-03-19T20:35:02.363Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34262","time":"2026-03-19T20:35:02.459Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:34266","time":"2026-03-19T20:35:02.582Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45334","time":"2026-03-19T20:35:02.940Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45344","time":"2026-03-19T20:35:03.488Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45358","time":"2026-03-19T20:35:04.269Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45360","time":"2026-03-19T20:35:04.971Z"}
{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45370","time":"2026-03-19T20:35:05.980Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45380","time":"2026-03-19T20:35:06.582Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45384","time":"2026-03-19T20:35:09.089Z"}
{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45398","time":"2026-03-19T20:35:11.277Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:45404","time":"2026-03-19T20:35:11.967Z"}
{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:43586","time":"2026-03-19T20:35:16.548Z"}
{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:43602","time":"2026-03-19T20:35:16.672Z"}
{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:43604","time":"2026-03-19T20:35:21.895Z"}

The router logs show this:

{"ctrlId":"","detail":{"endpoints":[{"address":"tls:ziti-controller.domain.com:443"}]},"file":"github.com/openziti/ziti/v2/router/env/ctrls.go:287","func":"github.com/openziti/ziti/v2/router/env.(*networkControllers).connectToControllerWithBackoff","level":"info","msg":"starting connection attempts","time":"2026-03-19T20:36:13.719Z"}
{"ctrlId":"","detail":{"endpoints":[{"address":"tls:ziti-controller.domain.com:443"}]},"endpoint":"tls:ziti-controller.domain.com:443","error":"error connecting ctrl (remote error: tls: internal error)","file":"github.com/openziti/ziti/v2/router/env/ctrls.go:282","func":"github.com/openziti/ziti/v2/router/env.(*networkControllers).connectToControllerWithBackoff.func1","level":"error","msg":"unable to connect controller","time":"2026-03-19T20:36:13.732Z"}

I can only presume that this is a networking issue, but I am not exactly spotting where the issue lies.

7

I tried again and spotted this in the controller logs just as I install the router chart:

{"_context":"tls:0.0.0.0:1280","error":"not handler for requested protocols [ziti-ctrl]","file":"github.com/openziti/transport/v2@v2.0.214/tls/listener.go:269","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.53:47336","time":"2026-03-20T02:53:59.890Z"}
8

For anyone who runs into this with the new charts/ziti versions, the solution for me was this:

ctrlPlane:
  # -- cluster service target port on the container
  containerPort: "{{ .Values.clientApi.containerPort }}"
  # -- global DNS name by which routers can resolve a reachable IP for this
  # service: default is cluster service DNS name which assumes all routers are
  # inside the same cluster
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  # -- cluster service, node port, load balancer, and ingress port
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  # -- besides advertisedHost, add these DNS SANs to the ctrl plane identity and any ctrl plane ingresses
  dnsNames: []
  service:
    # -- create a separate cluster service for the ctrl plane (default: disabled, shares listener with clientApi via ALPN)
    enabled: false

I previously had the ctrlPlane on a separate service, which worked, and I actually question why it functioned before.
Either way, collapsing the ctrlPlane service into the clientApi was the fix!

Thank you @qrkourier for the input!

1 Like
9

That should continue to work. Only the web and ctrl root CAs are deprecated, not their intermediates or leaves. In other words, now there's a single edge root CA, and all the other certs can trace their ancestry to that root.

10

With the depreciation notice in webBindingPki, what is the best way to move alternative certificates into edgeSignerPki?

11

Only the separate, additional root CAs for web and ctrl are deprecated. Their intermediates are now descended from the single, shared edge root CA, and so their intermediates continue to issue leaves in the same way, and any alternative issuers may still be configured in the same way for the web TLS listeners.