HA cluster behind NLB: enrollment token consumption fails on followers — forwarding not working for enrollment?

Ziti Overlay
1

First, thanks for making OpenZiti.

We're running a 3-node HA controller cluster (v2.0.0-pre10) behind an AWS NLB. All three nodes are healthy, fully connected, and Raft consensus is working correctly (ziti agent cluster list shows all three connected with a stable leader).

When an enrollment token is created and then consumed, if the NLB routes the consumption request to a follower, we get one of two behaviors:

  • 400 INVALID_ENROLLMENT_TOKEN immediately (fast fail)

  • 60 second timeout before failure

We expected follower controllers to forward write requests to the leader transparently, as described in the HA docs. This appears to work for other operations but not for enrollment token consumption.

We attempted to set edge.api.address to a per-node address on each controller so that inter-controller forwarding wouldn't loop back through the NLB. However this breaks OIDC login because edge.api.address also controls the OIDC issuer URL, which must match the address clients connect to via the NLB.

This is our config for each controller instance:


v: 3
cluster:
  dataDir: /var/lib/private/ziti-controller/raft

ctrl:
  listener: tls:0.0.0.0:6262
  options:
    advertiseAddress: tls:controller-1.<env>.example.net:6262

edge:
  api:
    address: controller.<env>.example.net:1280

web:
  - name: all-apis
    bindPoints:
      - interface: 0.0.0.0:1280
        address: controller.<env>.example.net:1280
    apis:
      - binding: health-checks
      - binding: fabric
      - binding: edge-management
      - binding: edge-client
      - binding: edge-oidc

We want to understand Is enrollment token consumption excluded from follower-to-leader forwarding. And is there a supported config pattern for separating the inter-controller forwarding address from the edge.api.address OIDC issuer URL in an NLB deployment?

2

Currently the tokens are signed using the controller's server cert/key pair and part of the bootstrapping process relies upon a TLS handshake w/ the controller that has the signing server cert/key.

Putting any kind of load balancer in-front of the controllers that terminates/offloads/transitions TLS will break the current enrollment processes.

There is a newer version of enrollment that has been put on the back burner that avoids this issue, but it is not currently released. When it is released, it will require SDK updates that will most likely take additional time.

Additionally, load balancers that terminates/offloads/transitions TLS will break certificate authentication as the controllers require mTLS for that authentication method. 2.0 also introduces certificate proof-of-possession, so mTLS used to initiate authentication (either due to certificate auth or auth-time CSR provisioning) will also break.

In general, load balancers that terminates/offloads/transitions TLS breaks mTLS, which is a central part of OpenZiti's fundamental security model.