Alternative DNS breaks tunneler connectivity

1

A Ziti controller configured without an alternative DNS, as per the helm chart values below, works as expected:

        clientApi:
          advertisedHost: "ziti-api.mydomain.com"
          advertisedPort: 443
          ingress:
            enabled: false
          service:
            type: ClusterIP
          traefikTcpRoute:
            enabled: true
        webBindingPki:
          enabled: false

Adding an alternative DNS name as below breaks the connection from a Linux tunneler:

        clientApi:
          advertisedHost: "ziti-api.mydomain.com"
          advertisedPort: 443
          altDnsNames:
            - "ziti-web.mydomain.com"
          ingress:
            enabled: false
          service:
            type: ClusterIP
          traefikTcpRoute:
            enabled: true
        webBindingPki:
          enabled: true
          altServerCerts:
            - mode: certManager
              secretName: ziti-controller-trusted-cert
              altDnsNames:
                - "ziti-web.mydomain.com"
              issuerRef:
                group: cert-manager.io
                kind: ClusterIssuer
                name: letsencrypt-prod
              mountPath: /etc/ziti/trusted-cert

In order for the tunneler to work, I have to reconfigure it to use the alternate DNS and add the Let's Encrypt root CA to the config file. I was expecting the connection to the advertizedHost to continue working.

Here's the output of the error log:

(4011277)[        0.009]    INFO ziti-sdk:ziti.c:512 ziti_start_internal() ztx[1] enabling Ziti Context
(4011277)[        0.009]    INFO ziti-sdk:ziti.c:528 ziti_start_internal() ztx[1] using tlsuv[v1.7.12/OpenSSL 3.4.2 1 Jul 2025]
(4011277)[        0.009]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-api.mydomain.com:443/] controller initialized
(4011277)[        0.009]    INFO ziti-sdk:ziti.c:606 ztx_init_controller() ztx[1] Loading ziti context with controller[https://ziti-api.mydomain.com:443/]
(4011277)[        0.468]   ERROR tlsuv:engine.c:868 openssl: handshake was terminated: SSL routines
(4011277)[        0.468]   ERROR tlsuv:tls_link.c:113 TLS(0x1babf120) handshake error certificate verify failed
(4011277)[        0.468]   ERROR tlsuv:http.c:192 handshake failed status[3]: certificate verify failed
(4011277)[        0.468]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-api.mydomain.com:443/] request failed: -103(software caused connection abort)
(4011277)[        0.468]    WARN ziti-sdk:ziti_ctrl.c:336 internal_version_cb() ctrl[https://ziti-api.mydomain.com:443/] CONTROLLER_UNAVAILABLE(software caused connection abort)
(4011277)[        0.468]    WARN ziti-sdk:ziti.c:2042 version_pre_auth_cb() ztx[1] failed to get controller version: CONTROLLER_UNAVAILABLE/software caused connection abort
(4011277)[        0.468]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ziti-api.mydomain.com:443/] request failed: -103(software caused connection abort)
(4011277)[        0.468]    INFO ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[https://ziti-api.mydomain.com:443/] attempting to switch endpoint
(4011277)[        0.468]    WARN ziti-sdk:ziti_ctrl.c:602 ctrl_next_ep() ctrl[https://ziti-api.mydomain.com:443/] no controllers are online
(4011277)[        0.468]    WARN ziti-sdk:ziti.c:643 ext_jwt_singers_cb() ztx[1] failed to get external auth providers: software caused connection abort
(4011277)[        5.880]   ERROR tlsuv:engine.c:868 openssl: handshake was terminated: SSL routines
(4011277)[        5.880]   ERROR tlsuv:tls_link.c:113 TLS(0x1babf120) handshake error certificate verify failed
(4011277)[        5.880]   ERROR tlsuv:http.c:192 handshake failed status[3]: certificate verify failed

I'll gladly log an issue, but before I do, I want to make sure my configuration is correct.

2

The tunneler should continue using the same advertised host, not switch to any alternative hosts for alternative server certs, which are only for Ziti-naive clients, such as a web browser, which rely on a default OS trust store, typically.

We need to figure out why "Adding an alternative DNS name as below breaks the connection from a Linux tunneler". There should be no ambiguity during server certificate selection as long as the advertised host and alt host are distinct, e.g., ziti-api.mydomain.com and ziti-web.mydomain.com. With this example, you tunneler should always use the ziti-api name, and you could access the web console on the ziti-web name to enjoy a publicly-trusted cert with HTTPS there.

The logs show only that the tunneler is continuing to use the ziti-api name, as it should, but encounters an unspecified TLS problem, or I'm unable to recognize it from the cryptic output.

You configuration looks fine at a glance, and I've certainly used that approach myself to bind a publicly trusted cert for the web listener for the console.

Let's first ensure the ziti-api path still exists after you redeploy the controller with the alt cert config. Hopefully, we can see what is the TLS problem the tunneler encountered.

openssl s_client -connect ziti-api.mydomain.com:443 -alpn h2,http/1.1 <>/dev/null

The HTTP ALPN is used by the tunneler to request the client API's protocol because, often, there are multiple protocols available on the same TCP port. For a controller's client port, ALPN ziti-ctrl is also tyically available.

3

I tested successfully with these values in my dev env. It's not usually necessary to override the chart's default app version, which is tested in tandem with the chart itself. Still, I wanted to ensure the problem couldn't be reproduced in the latest Ziti release, 1.7.0. I suspect the problem is a misalignment of these tricky input values. You can use any K8s service type or ingress controller that provides TLS passthrough.

clientApi:
  advertisedHost: miniziti-controller.192.168.58.2.sslip.io
  altDnsNames:
  - miniziti-controller.minikube1.bingnet.cloud
  ingress:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    enabled: true
    ingressClassName: nginx
  service:
    enabled: true
    type: ClusterIP
ctrlPlane:
  ingress:
    enabled: false
  service:
    enabled: false
image:
  additionalArgs:
  - --verbose
  repository: openziti/ziti-controller
  tag: 1.7.0
webBindingPki:
  altServerCerts:
  - altDnsNames:
    - miniziti-controller.minikube1.bingnet.cloud
    issuerRef:
      group: cert-manager.io
      kind: ClusterIssuer
      name: cloudflare-dns-issuer-prod
    mode: certManager
    mountPath: /etc/ziti/alt-server-cert
    secretName: ziti-controller-alt-server-cert1
4

This seems to work:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN=openziti-controller-web-identity
   i:CN=openziti-controller-web-intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Sep 23 12:18:50 2025 GMT; NotAfter: Oct  1 12:18:50 2035 GMT
 1 s:CN=openziti-controller-web-intermediate
   i:CN=openziti-controller-web-root
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Sep 23 12:18:47 2025 GMT; NotAfter: Oct  1 12:18:47 2035 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEdzCCBB2gAwIBAgIRALBIuZaErCfGvyJy+Cyyd5UwCgYIKoZIzj0EAwIwLzEt
MCsGA1UEAxMkb3BlbnppdGktY29udHJvbGxlci13ZWItaW50ZXJtZWRpYXRlMB4X
DTI1MDkyMzEyMTg1MFoXDTM1MTAwMTEyMTg1MFowKzEpMCcGA1UEAxMgb3Blbnpp
dGktY29udHJvbGxlci13ZWItaWRlbnRpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQDOgUe6JpvO/JzseMrfooq9KaDks5//gonmFrDgPVrlg2k78CeN
BnNM4hBq5fzMqwOQjdGIum9ph27lUd09uz0podl+sqNOCioPiJMpwFXBCY2IwqtB
MQJrWStq2ByIHtbyMTLRFzNfZpkYGVBTe9tqRVWWA536HNaGZsyUxluNgI6w/ftJ
ydx2mrW8tBOCaKuThbTn0BLj+uU3C3xlxN8IW7WYZGZsQWHUemwupRHx9Ut65yfw
QlfhYJwHR8w7Z2CURKfaCty5GW86ZUPvc34TMtX1jMB0EzbbzmInlFX4nDuf9LR1
FgyTdrGvjIdarb30kSIlYaD6Ho1fwMxyTEXPWWY+ib/31N1t0mt80ZX0DFVEb69Z
XjBSvn3F+07gjh1J6OY7/C28Ew/VaAT9uWkhGick9YRp5mK9J4Ug6njgYQB/wn4i
TUYu5M44wswdpVZnHN2UV6DLzB9Mx+NprZl5ny15YDQAbbskVcThGSAf6OAlqbNo
ZSEmbraw1ieVbJvbdSHFigkYxlRy2uRM94DETtLs86dx6+2rvZa342rtYkssbSRV
aQumtxHd6+ZEcee6owHTjrnHZnGF4Z3rfDpfsn0rXA7OmNTtszRa1yuovI4jPorn
fVSBnVFUU39I12A5dkO9szU/j8/y2YdNzPCC0ilDaeiv15zAxwCTaPwB6wIDAQAB
o4IBUTCCAU0wDgYDVR0PAQH/BAQDAgXgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwG
A1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUGmgpYwpq6qPjqcCNiAyMEvAu2DcwgfYG
A1UdEQSB7jCB64IJbG9jYWxob3N0gg96aXRpLWNvbnRyb2xsZXKCGm9wZW56aXRp
LWNvbnRyb2xsZXItY2xpZW50giNvcGVueml0aS1jb250cm9sbGVyLWNsaWVudC5v
cGVueml0aYInb3BlbnppdGktY29udHJvbGxlci1jbGllbnQub3BlbnppdGkuc3Zj
gjVvcGVueml0aS1jb250cm9sbGVyLWNsaWVudC5vcGVueml0aS5zdmMuY2x1c3Rl
ci5sb2NhbIIUeml0aS1hcGkuYWdpYmFzZS5jb22HBH8AAAGHEAAAAAAAAAAAAAAA
AAAAAAEwCgYIKoZIzj0EAwIDSAAwRQIgR4ImMYvkPDe6C9zRZnlfYyjzNECKRwqN
fVQDQvRyxncCIQCtwv8QlLrGW32SSjh21wdBJtJ89/1xpcHHhcxHoiWcaw==
-----END CERTIFICATE-----
subject=CN=openziti-controller-web-identity
issuer=CN=openziti-controller-web-intermediate
---
Acceptable client certificate CA names
CN=openziti-controller-edge-signer
CN=openziti-controller-web-root
CN=openziti-controller-ctrl-plane-root
CN=openziti-controller-edge-root
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2663 bytes and written 443 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 4096 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
ALPN protocol: http/1.1
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 921953DEB95579FA015D5CBE8E99DD26F0F23DC2E724870A38321122AEDE5E39
    Session-ID-ctx: 
    Resumption PSK: E757D249F4F45FBAA8740691EFA815A6F340D3A5FE2F7B54B84CC5B14A2F0748
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 59 ee fe 46 b5 44 c5 e8-e6 5e ad 48 b9 d1 09 0a   Y..F.D...^.H....
    0010 - 27 6e e8 d6 73 4c 11 25-b8 dc 79 9c 65 ea e5 af   'n..sL.%..y.e...
    0020 - 7b 54 4c a2 8e 66 31 87-ad e6 1c bc 55 ba 6c 88   {TL..f1.....U.l.
    0030 - 11 31 63 13 a5 f9 c1 5b-81 bc f4 f2 e0 63 38 4e   .1c....[.....c8N
    0040 - 4c d4 b9 43 fa a6 b0 27-cc 60 bb 73 5c 1d e1 e1   L..C...'.`.s\...
    0050 - f9 3e fb f2 d0 5e a2 28-86 40 ca 82 27 59 db b3   .>...^.(.@..'Y..
    0060 - 3f 9d f0 50 6c fb 8e 11-f0                        ?..Pl....

    Start Time: 1760701570
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

I modified the chart values to match what you shared:

clientApi:
  advertisedHost: "ziti-api.mydomain.com"
  advertisedPort: 443
  altDnsNames:
    - "ziti-web.mydomain.com"
  ingress:
    enabled: false
  service:
    type: ClusterIP
  traefikTcpRoute:
    enabled: true
ctrlPlane:
  ingress:
    enabled: false
  service:
    enabled: false
image:
  additionalArgs:
    - --verbose
  repository: openziti/ziti-controller
  tag: 1.7.0
webBindingPki:
  enabled: true
  altServerCerts:
    - mode: certManager
      secretName: ziti-controller-trusted-cert
      altDnsNames:
        - "ziti-web.mydomain.com"
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-prod
      mountPath: /etc/ziti/alt-server-cert

Unfortunately, I still get the same result. The controller logs show:

{"_context":"tls:0.0.0.0:1280","client":{"IP":"10.42.3.45","Port":42008,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.198/tls/listener.go:303","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = [http/1.1]","time":"2025-10-17T12:16:24.560Z"}
{"_context":"tls:0.0.0.0:1280","client":{"IP":"10.42.3.45","Port":42008,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.198/tls/listener.go:326","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2025-10-17T12:16:24.560Z"}
{"_context":"tls:0.0.0.0:1280","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.198/tls/listener.go:260","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.42.3.45:42008","time":"2025-10-17T12:16:24.765Z"}

The traefik ingressroute tcp is created with TLS passthrough:

apiVersion: v1
items:
- apiVersion: traefik.io/v1alpha1
  kind: IngressRouteTCP
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"traefik.io/v1alpha1","kind":"IngressRouteTCP","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"ziti-controller"},"name":"openziti-controller","namespace":"openziti"},"spec":{"entryPoints":["websecure"],"routes":[{"match":"HostSNI(`ziti-api.mydomain.com`)","services":[{"name":"openziti-controller-client","port":443}]}],"tls":{"passthrough":true}}}
    creationTimestamp: "2025-09-23T12:18:36Z"
    generation: 10
    labels:
      argocd.argoproj.io/instance: ziti-controller
    name: openziti-controller
    namespace: openziti
    resourceVersion: "270592960"
    uid: d091b0bf-62e0-4808-b27f-3a75251ae60c
  spec:
    entryPoints:
    - websecure
    routes:
    - match: HostSNI(`ziti-api.mydomain.com`)
      services:
      - name: openziti-controller-client
        port: 443
    tls:
      passthrough: true