Alternate server certificate not selected, workaround

Support General Questions
1

Greetings,

Edit 2026-02-09 (workaround):

I stopped the zac container and installed the console into the controller container using this guide https://netfoundry.io/docs/openziti/learn/quickstarts/zac/.

Now the correct certificate is returned. The console was accessed at :1280 instead of the :9443 I was using for the zac container. (:8443 was already in use.)

This seems to be a successful workaround. A minor issue is that the Brave browser reports the connection as not secure despite receiving the correct valid certificate and despite connecting using https. The Firefox and Chromium browser both report a secure connection.

Using the Docker Quickstart, I have a controller, router, and console up and running in separate containers. Everything seems to be running correctly but despite pouring over the available docs and messages in this forum, I cannot get the console alternate server certificate to be selected. Can anyone give me an idea what to check next?

I run a split horizon using Bind for local DNS. I have a wildcard certificate from Let's Encrypt that I use for both public and private nets. Since my public certificate is a wildcard I was afraid that just adding a subdomain to mydomain.pubtld for OpenZiti would be problematic one way or another so I set up a separate internal zone and subnet that does not overlap my public/private domain at all.

    Public/Private Cert              Ziti Generated Cert
    ---------------------------      ------------------------------
    SAN:   *.mydomain.pubtld         SAN: edge.zdom.pritld
    SAN:    mydomain.pubtld          SAN: localhost
                                     SAN: zdom
                                     SAN: IP Address: 127.0.0.1

The machine hosting the docker containers can be reached at:

    testhost.mydomain.pubtld            ctrl.zdom.pritld
                                        edge.zdom.pritld
                                        er.zdom.pritld
                                        zac.zdom.pritld

Attempting to acccess the console at testhost.mydomain.pubtld does not select the mydomain.pubtld certificate. Instead the ziti generated certificate is returned (along with accompanying browser security complaint).

I have checked:

  1. Using wireshark I verified that the TLS SNI is in fact testhost.mydomain.pubtld as expected.
  2. As shown above, there is no overlap in the SANs.
  3. The alt server cert was configured on the same web listener as the edge management API.
  4. I uncommented the zac binding section in the same web listener section as described in (3)
    1. I am not at all sure if the default values here are correct but being on separate containers
      I am not sure what would be correct      .       The default config seems to reference a location option that does not exist. (Controller config copied below.)
    2. I also tried leaving the zac binding commented out.
  5. Verified that the alt server files are readable inside the controller container by
    a) Catting the cert and key at the cli.
    b) Changing permissions on the host folder and observing a panic in the container logs
  6. Haven’t found any explanation in logs.
  7. My certificate files are .cer’s but I verified they are pem-encoded.

Thank you for any recommendations offered!

Controller configuration:

...

identity:
  cert:        "/persistent/pki/ctrl.zdom.pritld-intermediate/certs/ctrl.zdom.pritld-client.chain.pem"
  server_cert: "/persistent/pki/ctrl.zdom.pritld-intermediate/certs/ctrl.zdom.pritld-server.chain.pem"
  key:         "/persistent/pki/ctrl.zdom.pritld-intermediate/keys/ctrl.zdom.pritld-server.key"
  ca:          "/persistent/pki/cas.pem"

ctrl:
  options:
    advertiseAddress: tls:ctrl.zdom.pritld:6262
  listener:             tls:0.0.0.0:6262

healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s

edge:
  api:
    sessionTimeout: 30m
    address: edge.zdom.pritld:1280
  enrollment:
    signingCert:
      cert: /persistent/pki/signing.pem
      key:  /persistent/pki/ziti-signing-intermediate/keys/ziti-signing-intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:1280
        address: edge.zdom.pritld:1280
    identity:
      ca:          "/persistent/pki/edge.zdom.pritld-root-ca/certs/edge.zdom.pritld-root-ca.cert"
      key:         "/persistent/pki/edge.zdom.pritld-intermediate/keys/edge.zdom.pritld-server.key"
      server_cert: "/persistent/pki/edge.zdom.pritld-intermediate/certs/edge.zdom.pritld-server.chain.pem"
      cert:        "/persistent/pki/edge.zdom.pritld-intermediate/certs/edge.zdom.pritld-client.chain.pem"
      alt_server_certs:
      - server_cert: "/etc/zac_alt_server_certs/fullchain.cer"
        server_key:  "/etc/zac_alt_server_certs/mydomain.pubtld.key"
      
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-oidc
        options: { }
      - binding: zac                        <<< ???
        options:
          location: ./console
          indexFile: index.html
2

Hi @shawncp welcome to the community and to OpenZiti! Sorry for the delay here, it's been a busy time... Thanks for your patience.

Generally speaking when alt certs are not presented it has historically been because the alternative certificate overlaps with the actual controller url.

When the quickstart starts and you give it an "external dns" if that entry you provide matches the alt cert, you'll have non-deterministic behavior and it can be hard to spot. Could that possibly be what's happened?

For example when I set my controller up, I use two very different FQDNs. The free one that AWS gives me I use for my overlay (EXTERNAL_DNS) https://ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441

and then I get a LetsEncrypt cert for another FQDN at https://ctrl.cdaws.clint.demo.openziti.org:8441

Is that maybe the problem here?

3

AAAnnnnd then i reread your post but with all the detail you provided (and thank you for that) i saw this: " does not overlap my public/private "

So... ok. I'll go back and reread it all again and try not to skip over important detail... :confused:

Ok I read back and see an update was posted too. That didn't make sense at first but I get it now that I read (and commented) and reread.

Sounds like you got thing sorted out other than Brave. If I can help any more lemme know. Sorry it didn't click the first time I read the post cheers!

4

Hi Lumberjack,

Thank you for you time and comments.

Yes the workaround I described was successful. I still have not figured out why the console-in-a-separate-container did not behave correctly so in that sense this is still unsolved and I’m still puzzled because I must have missed something, just cannot figure what. To add to the mystery, installing the console in the controller container instantly worked correctly with no other change to configuration.

Anyway, having the console in the controller container seems fine. Thanks again.

5

Oh it just came to me. The short comment about what might be different when running in a separate container triggered it in my brain...

When you run the console in a separate container, you're running a separate node server, entirely separate from the controller... That's the difference. The compose file is what you would update to point to the correct certs then. It wouldn't be the controller configuration at all. You would instead copy the letsencrypt certs into that contain (or mount them, etc)

All that said, mounting the zac in the controller has been our recommended approach for a while and is definitely a recommended deployment configuration.

Anyway, i just wanted to close this gap on this issue, since the actual answer occurred to me! Cheers