Alt_server_certs in HA mode enrolment failed

1

Hi there,

I'm using Ziti Controllers in HA mode v1.5.4 installed locally on my network.

I want the Controller HTTP API's to present a publicly trusted certificate but use a private PKI with different domain for everything else. Signing identity certs, Overlay traffic ETC. My private PKI uses a completely separate domain. E.G lifeboat.ziti

For example, I've created a Lets Encrypt certificate for *.ziti.example.com and configured it's server_cert and server_key file in web.identity.alt_server_certs. Now when i curl https://ziti-controller-1.ziti.example.com the presented certificate is trusted and i no longer get self signed cert error. ZAC also presents the publicly trusted cert as long as i address the service using the ziti.example.com domain.

My Controllers and Edge routers all connect to each other no problem.

However when i attempt to enrol an identity using ZET, i get TLS errors below.

Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-07-03T15:58:10.652)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-07-03T15:58:10.652)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.ziti.example.com:443] controller initialized
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.ziti.example.com:443] controller initialized
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]   ERROR tlsuv:engine.c:955 openssl: handshake was terminated: error:00000005:lib(0)::reason(5)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]   ERROR tlsuv:tls_link.c:113 TLS(0x559e6c7fa0e0) handshake error error:00000005:lib(0)::reason(5)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]   ERROR tlsuv:http.c:188 handshake failed status[3]: error:00000005:lib(0)::reason(5)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] request failed: -103(software caused connection abort)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    WARN ziti-sdk:ziti_ctrl.c:336 internal_version_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] CONTROLLER_UNAVAILABLE(software caused connection abort)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] request failed: -103(software caused connection abort)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] attempting to switch endpoint
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]    WARN ziti-sdk:ziti_ctrl.c:602 ctrl_next_ep() ctrl[https://ziti-controller-1.ziti.example.com:443] no controllers are online
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]   ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://ziti-controller-1.ziti.example.com:443 CONTROLLER_UNAVAILABLE[software caused connection abort] reason[]
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[        0.000]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: ziti controller is not available(-16)
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5169]: Segmentation fault
Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5169]: ERROR: failed to enroll zet-fcjgbtpr.lifeboat.controller.jwt in /opt/openziti/etc/identities

Controller log

Jul 03 15:58:10 ziti-controller-1 ziti[2712]: {"_context":"tls:0.0.0.0:443","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"192.168.54.118:48984","time":"2025-07-03T15:58:10.666Z"}

Controller config

v: 3

commandRateLimiter:
  enabled:   true
  maxQueued: 1000

tls:
  handshakeTimeout: 30s
  rateLimiter:
    enabled: true
    minSize: 5
    maxSize: 250

cluster:
  dataDir: ./data/

identity:
  cert: ./pki/ziti-controller-1/certs/server.chain.pem
  key: ./pki/ziti-controller-1/keys/server.key
  ca: ./pki/ziti-controller-1/certs/ziti-controller-1.chain.pem

ctrl:
  listener: tls:0.0.0.0:8443
  options:
    advertiseAddress: tls:ziti-controller-1.lifeboat.ziti:8443
    maxQueuedConnects: 100
    maxOutstandingConnects: 100
    connectTimeoutMs: 2000
    writeTimeout: 15s

events:
  jsonLogger:
    subscriptions:
      - type: connect
      - type: cluster
    handler:
      type: file
      format: json
      path: /tmp/ziti-events.log

edge:
  api:
    address: ziti-controller-1.ziti.example.com:443
  enrollment:
    signingCert:
      cert: ./pki/ziti-controller-1/certs/ziti-controller-1.cert
      key: ./pki/ziti-controller-1/keys/ziti-controller-1.key
    edgeIdentity:
      duration: 600m
    edgeRouter:
      duration: 10m

web:
  - name: all-apis-localhost
    identity:
      cert: ./pki/ziti-controller-1/certs/ziti-controller-1.cert
      key: ./pki/ziti-controller-1/keys/ziti-controller-1.key
      server_cert: ./pki/ziti-controller-1/certs/server.chain.pem
      server_key: ./pki/ziti-controller-1/keys/server.key
      ca: ./pki/ziti-controller-1/certs/ziti-controller-1.chain.pem
      alt_server_certs:
        - server_cert: ./pki/alt/lets_encrypt.cert
          server_key: ./pki/alt/lets_encrypt.key
    bindPoints:
      - interface: 0.0.0.0:443
        address: ziti-controller-1.ziti.example.com:443
    options:
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: health-checks
      - binding: fabric
      - binding: edge-management
      - binding: edge-client
      - binding: edge-oidc
      - binding: zac
        options:
          location: /opt/openziti/share/console
          indexFile: index.html
2

Hi @farmhouse, it's always tricky getting this right but I think the issue is in the web api and advertising the public cert for your apis. Using a "public" cert is fine for the zac, but all the trust around enrollment will be broken doing this.

Instead, just change

        address: ziti-controller-1.ziti.example.com:443

back to

        address: ziti-controller-1.lifeboat.ziti:443

You never need to 'advertise' the LE certs anywhere. You simply need to access your server at that FQDN.

For example my LE-backed server: https://ctrl.cdaws.clint.demo.openziti.org:8441/zac/dashboard

Advertises the private PKI:

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441

I think that's your whole problem. If you want to understand why or learn more I can explain, but if you want to "just fix it" that's all you need to do I think

You'll still get a publicly trusted CA chain, you just don't need to advertise it.

curl -s -X GET -I https://ctrl.cdaws.clint.demo.openziti.org:8441/version | head -1
HTTP/1.1 200 OK
3

Thanks @TheLumberjack i will do some more testing tomorrow based on your feedback.

Is it actually possible to present the publicly trusted cert on the edge-client API using the alt_server_certs method while continuing to use private PKI for enrollment or is alt_server_certs really only for ZAC and the management API Etc ?

Im not at my workstation right now but i'm sure i got an error in the Controller when edge.api.address and web.bindPoints.address are not the same.

4

OpenZiti uses explicit trust, and all the components go through a strict process of bootstrapping that trust so they all rely on the private PKI.

Is it actually possible to present the publicly trusted cert on the edge-client API

I know of no reason to do this in particular. It's really ONLY necessary for ZAC and potentially for the management API. It's never needed for the client API. The client API will have trust bootstrapped by whatever identity is accessing it.

One reason you might want to do it for the management API is if you are integrating with OpenZiti's controller and don't want to pull the cert bundle from the controller (which you can obtain from ${your.controller}/.well-known/est/cacerts. for example https://ctrl.cdaws.clint.demo.openziti.org:8441/.well-known/est/cacerts. I will often use the ca bundle with curl if i care (usually I'm doing development stuff so curl's -k is fine for me).

Lastly, if you have OIDC integration and you want to add an identity by URL, you MUST have alt server certs setup. In this case, you would be distributing a URL, not a JWT and that's a big, big difference. A JWT is a signed document, a URL is not. As such, the URL needs to be trusted somehow and that's where a publicly trusted CA would be mandatory.

You're right, I missed this in your config:

  api:
    address: ziti-controller-1.ziti.example.com:443

You need to update that to not be the 'public' fqdn as well. :slight_smile: The public FQDN never needs to appear in your config file. Cheers