# Alt\_server\_certs in HA mode enrolment failed **URL:** \ **Category:** Uncategorized\ **Created:** [July 3, 2025, 4:05pm UTC](https://openziti.discourse.group/t/alt-server-certs-in-ha-mode-enrolment-failed/4844 "2025-07-03T16:05:27Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![farmhouse](https://avatars.discourse-cdn.com/v4/letter/f/b19c9b/32.png) [@farmhouse](https://openziti.discourse.group/u/farmhouse)\ **Post date:** [July 3, 2025, 4:05pm UTC](https://openziti.discourse.group/t/alt-server-certs-in-ha-mode-enrolment-failed/4844/1 "2025-07-03T16:05:27Z")
Hi there, I'm using Ziti Controllers in HA mode `v1.5.4` installed locally on my network. I want the Controller HTTP API's to present a publicly trusted certificate but use a private PKI with different domain for everything else. Signing identity certs, Overlay traffic ETC. My private PKI uses a completely separate domain. E.G `lifeboat.ziti` For example, I've created a Lets Encrypt certificate for `*.ziti.example.com` and configured it's `server_cert` and `server_key` file in `web.identity.alt_server_certs`. Now when i `curl https://ziti-controller-1.ziti.example.com` the presented certificate is trusted and i no longer get self signed cert error. ZAC also presents the publicly trusted cert as long as i address the service using the `ziti.example.com` domain. My Controllers and Edge routers all connect to each other no problem. However when i attempt to enrol an identity using ZET, i get TLS errors below. ```auto Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-07-03T15:58:10.652) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-07-03T15:58:10.652) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.ziti.example.com:443] controller initialized Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.ziti.example.com:443] controller initialized Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] ERROR tlsuv:engine.c:955 openssl: handshake was terminated: error:00000005:lib(0)::reason(5) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] ERROR tlsuv:tls_link.c:113 TLS(0x559e6c7fa0e0) handshake error error:00000005:lib(0)::reason(5) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] ERROR tlsuv:http.c:188 handshake failed status[3]: error:00000005:lib(0)::reason(5) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] request failed: -103(software caused connection abort) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] WARN ziti-sdk:ziti_ctrl.c:336 internal_version_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] CONTROLLER_UNAVAILABLE(software caused connection abort) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] request failed: -103(software caused connection abort) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] INFO ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[https://ziti-controller-1.ziti.example.com:443] attempting to switch endpoint Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] WARN ziti-sdk:ziti_ctrl.c:602 ctrl_next_ep() ctrl[https://ziti-controller-1.ziti.example.com:443] no controllers are online Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://ziti-controller-1.ziti.example.com:443 CONTROLLER_UNAVAILABLE[software caused connection abort] reason[] Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5173]: (5173)[0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: ziti controller is not available(-16) Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5169]: Segmentation fault Jul 03 15:58:10 zet-fcjgbtpr ziti-edge-tunnel.sh[5169]: ERROR: failed to enroll zet-fcjgbtpr.lifeboat.controller.jwt in /opt/openziti/etc/identities ``` Controller log ```auto Jul 03 15:58:10 ziti-controller-1 ziti[2712]: {"_context":"tls:0.0.0.0:443","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"192.168.54.118:48984","time":"2025-07-03T15:58:10.666Z"} ``` Controller config ```auto v: 3 commandRateLimiter: enabled: true maxQueued: 1000 tls: handshakeTimeout: 30s rateLimiter: enabled: true minSize: 5 maxSize: 250 cluster: dataDir: ./data/ identity: cert: ./pki/ziti-controller-1/certs/server.chain.pem key: ./pki/ziti-controller-1/keys/server.key ca: ./pki/ziti-controller-1/certs/ziti-controller-1.chain.pem ctrl: listener: tls:0.0.0.0:8443 options: advertiseAddress: tls:ziti-controller-1.lifeboat.ziti:8443 maxQueuedConnects: 100 maxOutstandingConnects: 100 connectTimeoutMs: 2000 writeTimeout: 15s events: jsonLogger: subscriptions: - type: connect - type: cluster handler: type: file format: json path: /tmp/ziti-events.log edge: api: address: ziti-controller-1.ziti.example.com:443 enrollment: signingCert: cert: ./pki/ziti-controller-1/certs/ziti-controller-1.cert key: ./pki/ziti-controller-1/keys/ziti-controller-1.key edgeIdentity: duration: 600m edgeRouter: duration: 10m web: - name: all-apis-localhost identity: cert: ./pki/ziti-controller-1/certs/ziti-controller-1.cert key: ./pki/ziti-controller-1/keys/ziti-controller-1.key server_cert: ./pki/ziti-controller-1/certs/server.chain.pem server_key: ./pki/ziti-controller-1/keys/server.key ca: ./pki/ziti-controller-1/certs/ziti-controller-1.chain.pem alt_server_certs: - server_cert: ./pki/alt/lets_encrypt.cert server_key: ./pki/alt/lets_encrypt.key bindPoints: - interface: 0.0.0.0:443 address: ziti-controller-1.ziti.example.com:443 options: minTLSVersion: TLS1.2 maxTLSVersion: TLS1.3 apis: - binding: health-checks - binding: fabric - binding: edge-management - binding: edge-client - binding: edge-oidc - binding: zac options: location: /opt/openziti/share/console indexFile: index.html ``` ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [July 3, 2025, 4:32pm UTC](https://openziti.discourse.group/t/alt-server-certs-in-ha-mode-enrolment-failed/4844/2 "2025-07-03T16:32:33Z")
Hi @farmhouse, it's always tricky getting this right but I think the issue is in the web api and advertising the public cert for your apis. Using a "public" cert is fine for the zac, but all the trust around enrollment will be broken doing this. Instead, just change ```auto address: ziti-controller-1.ziti.example.com:443 ``` back to ```auto address: ziti-controller-1.lifeboat.ziti:443 ``` You never need to 'advertise' the LE certs anywhere. You simply need to access your server at that FQDN. For example my LE-backed server: [https://ctrl.cdaws.clint.demo.openziti.org:8441/zac/dashboard](https://ctrl.cdaws.clint.demo.openziti.org:8441/zac/dashboard) Advertises the private PKI: ```auto web: - name: client-management bindPoints: - interface: 0.0.0.0:8441 address: ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441 ``` I think that's your whole problem. If you want to understand why or learn more I can explain, but if you want to "just fix it" that's all you need to do I think You'll still get a publicly trusted CA chain, you just don't need to advertise it. ```auto curl -s -X GET -I https://ctrl.cdaws.clint.demo.openziti.org:8441/version | head -1 HTTP/1.1 200 OK ``` ---
**Author:** ![farmhouse](https://avatars.discourse-cdn.com/v4/letter/f/b19c9b/32.png) [@farmhouse](https://openziti.discourse.group/u/farmhouse)\ **Post date:** [July 3, 2025, 7:47pm UTC](https://openziti.discourse.group/t/alt-server-certs-in-ha-mode-enrolment-failed/4844/3 "2025-07-03T19:47:56Z")
Thanks @TheLumberjack i will do some more testing tomorrow based on your feedback. Is it actually possible to present the publicly trusted cert on the edge-client API using the `alt_server_certs` method while continuing to use private PKI for enrollment or is `alt_server_certs` really only for ZAC and the management API Etc ? Im not at my workstation right now but i'm sure i got an error in the Controller when `edge.api.address` and `web.bindPoints.address` are not the same. ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [July 3, 2025, 8:05pm UTC](https://openziti.discourse.group/t/alt-server-certs-in-ha-mode-enrolment-failed/4844/4 "2025-07-03T20:05:29Z")
OpenZiti uses explicit trust, and all the components go through a strict process of bootstrapping that trust so they all rely on the private PKI. > Is it actually possible to present the publicly trusted cert on the edge-client API I know of no reason to do this in particular. It's really ONLY necessary for ZAC and _potentially_ for the management API. It's never needed for the client API. The client API will have trust bootstrapped by whatever identity is accessing it. One reason you **might** want to do it for the management API is if you are integrating with OpenZiti's controller and don't want to pull the cert bundle from the controller (which you can obtain from ${your.controller}/.well-known/est/cacerts. for example [https://ctrl.cdaws.clint.demo.openziti.org:8441/.well-known/est/cacerts](https://ctrl.cdaws.clint.demo.openziti.org:8441/.well-known/est/cacerts). I will often use the ca bundle with curl if i care (usually I'm doing development stuff so curl's `-k` is fine for me). Lastly, if you have OIDC integration and you want to add an identity by URL, you MUST have alt server certs setup. In this case, you would be distributing a URL, not a JWT and that's a big, big difference. A JWT is a signed document, a URL is not. As such, the URL needs to be trusted **somehow** and that's where a publicly trusted CA would be mandatory. You're right, I missed this in your config: ```auto api: address: ziti-controller-1.ziti.example.com:443 ``` You need to update that to not be the 'public' fqdn as well. 🙂 The public FQDN never needs to appear in your config file. Cheers