Need guidance on upgrding ziti-controller

Support
1

I am currently running a production setup on Kubernetes with the following versions:

  • Ziti Controller: 0.34.1
  • Ziti Router: 1.0.0
  • Ziti ZAC: 3.4.5

I need to upgrade the controller and would like some guidance before proceeding. My main concern is ensuring that the identities are not affected during the upgrade. If I perform a direct upgrade of the controller, will it impact the existing identities? Are there any dependencies between the controller, router, edge tunneler, and identities that I should be aware of before upgrading?

2

Continuity

I am looking at these versions now and will attempt to predict any significant issues you might encounter.

You asked specifically about identities. I would expand that concern to include all enrollments, and I can confirm that the proposed upgrade path from current to latest does not invalidate enrollments.

Disaster Recovery

I'll re-focus on continuity in a follow-up post, but having a disaster recovery plan for the controller is essential for production. As a reference point, here's the Linux controller backup guide. In short, ensure you can recover the PKI and database. Ziti routers and identities can be re-enrolled even if all data is catastrophically lost, albeit with an interruption. It's also a good idea to back up the router configurations, especially if they are intricately customized.

You're deploying the Ziti controller in Kubernetes, so Cert Manager manages the PKI, and a storage volume persists the database. You need to ensure that you have a way to restore those, at least, and the Helm release's inputs that feed into the controller's configuration are important, too.

I recommend periodically taking database snapshots and either offloading and backing up the snapshots or making volume snapshots that include the current DB and your retained DB snapshots in the same volume.

3

For future reference, you can review the changelog for the intervening versions including current and planned upgrade version. I found zero breaking change announcements between your current version 0.34.1, which was incidentally the final release before 1.0.0, and 1.1.9 which is the ziti version used by the latest chart.

The next question is "What significant changes have occurred for Helm charts since the chart that provided ziti controller 0.34.1?"

This is more complex, but here are some highlights from the diff:

  • the console chart is deprecated by a built-in console in the controller deployment - the console URL is printed after helm upgrade
  • the controller chart has an optional "Prometheus Operated" ServiceMonitor resource available
  • ziti networks require a trust domain: set .Values.trustDomain or a random value will be generated and stored in a secret
4

Okay, so if I properly back up the PKI and database files, I assume that the Helm upgrade will retrieve the files from the volume. This shouldn't affect the identities, and there won't be a need for re-enrollment. The reason I'm asking is that I have a device set up as an identity at a client location, and if that breaks, I wouldn't be able to perform a remote re-enrollment. I would have to physically visit the client location to fix it.

5

Correct. The latest controller chart will use the same PVC containing the database and the same PKI from Cert Manager.

You would have to do something destructive to invalidate that remote identity's enrollment, e.g.,

  • change the root CA
  • delete the identity
  • initiate re-enrollment of the identity (replaces client cert authenticator with pending enrollment)
6

Got it, So upgrade should not affect my identities. I can just backup for DR scenario and do helm upgrade.

No need to reenroll identities and routers right?

7

Correct. Your identity and router enrollments live in the BBolt DB file in the PVC and persist during a regular Helm upgrade operation. No need to re-enroll following an upgrade.

8

Hi @qrkourier , I have one more doubt

this is my controller yaml file

# /tmp/controller-values.yml
ctrlPlane:
    advertisedHost: ziti-ctrl.zzz.zzz
    advertisedPort: 443
    service:
        type: ClusterIP
    ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
            kubernetes.io/ingress.allow-http: "false"
            nginx.ingress.kubernetes.io/ssl-passthrough: "true"
            nginx.ingress.kubernetes.io/secure-backends: "true"
clientApi:
    advertisedHost: ziti-controller.zzzz.zzz
    advertisedPort: 443
    service:
        type: ClusterIP
    ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
            kubernetes.io/ingress.allow-http: "false"
            nginx.ingress.kubernetes.io/ssl-passthrough: "true"
            nginx.ingress.kubernetes.io/secure-backends: "true"

this is my router yaml file

ctrl:
  endpoint: ziti-ctrl.zzz.zzz:443
edge:
  advertisedHost: ziti-router.xxx.xxx
  advertisedPort: 443
  service:
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
linkListeners:
  transport:
    advertisedHost: ziti-router-tp.xxx.xxx
    advertisedPort: 443
    service:
      enabled: true
      type: ClusterIP
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        kubernetes.io/ingress.allow-http: "false"
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"

i think im missing letencrypt tls certificate here, how do i i add it? if i add tls certificate then will it break my identities i hope it wont, just wanted to be sure not to break the identies production before i upgrade.

9

Will you say more about how you plan to leverage a trusted certificate from a public CA like LetsEncrypt?

A trusted certificate is not always needed for a ziti controller. Instead of a trusted public CA, Ziti enrollments configure explicit trust for identities, routers, and controllers.

You might be thinking about one of these cases:

  1. a trusted server certificate for the web console
  2. a trusted server certificate for BrowZer clients
10

OK cool, then it should be fine what i have now.
I have CA Certificate for Webconsole already. so that should be fine.
Thanks

11

I have upgraded ziti-controller without any issues. thanks for the support.

1 Like