K8S controller pod crashes with webBindingPki enabled

Ziti Overlay
1

Hello, I am getting back to my project using openziti, this is new green fields deployment.

Congrats on version 2.0 :slight_smile: yay!

My Environment:

  1. K3S
  2. OpenZiti helm chart version: ziti-controller-3.2.0
  3. Openziti version 2.0
  4. Openziti standalone controller and 3x routers deployed via ansible and helm
  5. Ctrl, client and management listeners are on separate cluster IP addresses that are issued by kube-virt
  6. OIDC is not configured (well I have not configured it, yet)
  7. OIDC endpoint will be Authentik with trusted cert chain (not configured yet)

GOAL:

To only have the client listener exposed publicly for standard jwt and OIDC enabled laptops keeping everything else non exposed if possible.

So I might have some of the config wrong at the moment. I am doing baby steps to test and evolve.

REALITITY

When deploying the controller without webBindingPki configured, the environment come up and is stable. :smile:

However I need to add trusted alt_server_certs to the client endpoint, so that I can enable OIDC for my desktop tunnelers.

When I enable webBindingPki from a K8S secret with the ansible task,

webBindingPki:
  altServerCerts:
  - mode: secret
    secretName: corporate-web-certs

The certs are mounted into the pod and testing the cert chain, the chain is valid.

I notice in the crashing controller pod, that the controller is trying to bootstrap OIDC endpoints i.e /authorise etc (looping) and then failing with this error:
{"file":"github.com/openziti/xweb/v3@v3.0.4/server.go:101","func":"github.com/openziti/xweb/v3.NewServer","level":"fatal","msg":"encountered error building handler for api binding [edge-oidc]: could not create OpenIdProvider: invalid url for issuer"}

This is my rendered configmap from within the crashing pod with webBindingPki enabled:
ziti-controller.yaml:

---
v: 3
db:                     /persistent/ctrl.db
trustDomain:            ${ZITI_TRUST_DOMAIN}

identity:
  cert:                 /etc/ziti/ctrl-plane-client-identity/tls.crt
  key:                  /etc/ziti/ctrl-plane-client-identity/tls.key
  server_cert:          /etc/ziti/ctrl-plane-identity/tls.crt
  server_key:           /etc/ziti/ctrl-plane-identity/tls.key
  ca:                   ${ZITI_CTRL_PLANE_CA}/ctrl-plane-cas.crt

network:
  createCircuitRetries: 2
  cycleSeconds: 15
  initialLinkLatency: 65s
  minRouterCost: 10
  pendingLinkTimeoutSeconds: 10
  routeTimeoutSeconds: 10
  routerConnectChurnLimit: 1m
  smart:
    rerouteCap: 4
    rerouteFraction: 0.02

ctrl:
  listener:                tls:0.0.0.0:1280
  options:
    advertiseAddress: tls:ctrl.mesh.services:1280

healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s

edge:
  api:
    sessionTimeout: 30m
    address: client.mesh.services:1280

  enrollment:
    signingCert:
      cert: /etc/ziti/edge-signer/tls.crt
      key:  /etc/ziti/edge-signer/tls.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m

web:
  - name: client
    bindPoints:
      - interface: 0.0.0.0:1280
        address: client.mesh.services:1280
    identity:
      cert:        /etc/ziti/web-client-identity/tls.crt
      key:         /etc/ziti/web-client-identity/tls.key
      server_cert: /etc/ziti/web-identity/tls.crt
      server_key:  /etc/ziti/web-identity/tls.key
      ca:          /etc/ziti/web-identity/ca.crt
      alt_server_certs:
        - server_cert: "/etc/openziti/custom-certs/tls.crt"
          server_key:  "/etc/openziti/custom-certs/tls.key"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      # minTLSVersion - optional, default TLS1.2
      # The minimum version of TSL to support
      minTLSVersion: TLS1.2
      # maxTLSVersion - optional, default TLS1.3
      # The maximum version of TSL to support
      maxTLSVersion: TLS1.3
    # apis - required
    # Allows one or more APIs to be bound to this webListener
    apis:
      - binding: edge-client
        options: { }
      - binding: edge-oidc
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:1281
        address: management.mesh.services:1281
    identity:
      cert:        /etc/ziti/web-client-identity/tls.crt
      key:         /etc/ziti/web-client-identity/tls.key
      server_cert: /etc/ziti/web-mgmt-api/tls.crt
      server_key:  /etc/ziti/web-mgmt-api/tls.key
      ca:          /etc/ziti/web-mgmt-api/ca.crt
      alt_server_certs:
        - server_cert: "/etc/openziti/custom-certs/tls.crt"
          server_key:  "/etc/openziti/custom-certs/tls.key"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      # minTLSVersion - optional, default TLS1.2
      # The minimum version of TSL to support
      minTLSVersion: TLS1.2
      # maxTLSVersion - optional, default TLS1.3
      # The maximum version of TSL to support
      maxTLSVersion: TLS1.3
    # apis - required
    # Allows one or more APIs to be bound to this webListener
    apis:
      - binding: edge-client
        options: { }
      - binding: edge-oidc
        options: { }
      - binding: edge-management
        options: { }
      - binding: fabric
        options: { }
      - binding: health-checks
        options: { }
      - binding: zac
        options:
          location: /ziti-console
          indexFile: index.html

Commenting out the webBindingPki and redeploying the controller with the ansible rebuilds the configmap without

      alt_server_certs:
        - server_cert: "/etc/openziti/custom-certs/tls.crt"
          server_key:  "/etc/openziti/custom-certs/tls.key"

And it is back running again.

The tls cert and key is test using openssl within the pod and they are trusted at an os level.

I have this working in a POC using docker, but K8S and Helm is a new level.

Any assistance would be amazing.

Thank you for your time.