Haproxy + alt_server_certs

Support
1

I trying to setup alt_server_certs for OIDC, but having trouble.

I use haproxy proxy front of controller/router and my haproxy configs is:

frontend tcp_https_in
    bind *:443
    bind [::]:443 v6only
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    # This for controller
    use_backend controller_backend if { req_ssl_sni -i  controller1.domain.com }
    # This for OIDC
    use_backend controller_backend if { req_ssl_sni -i  ziti.domain.com }
    # This for router1
    use_backend edge_router_backend if { req_ssl_sni -i router1.domain.com

frontend tcp_https_in_8443
    bind *:8443
    bind [::]:8443 v6only
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    # This for controller
    use_backend controller_backend_8443 if { req_ssl_sni -i  controller1.domain.com }

backend controller_backend
    mode tcp
    server controller 127.0.0.1:1280 check

backend edge_router_backend
    mode tcp
    server edge_router 127.0.0.1:3022 check

backend controller_backend_8443
    mode tcp
    server controller 127.0.0.1:1282 check

And in controller config I have

identity:
  cert:        "pki/intermediate/certs/client.chain.pem"
  server_cert: "pki/intermediate/certs/server.chain.pem"
  key:         "pki/intermediate/keys/server.key"
  ca:          "pki/root/certs/root.cert"
  alt_server_certs:
  - server_cert: "/etc/letsencrypt/live/ziti.domain.com/fullchain.pem "
    server_key:  "/etc/letsencrypt/live/ziti.domain.com/privkey.pem"

...

web:
  - name: client-management
    bindPoints:
      - interface: 127.0.0.1:1280
        address: controller1.domain.com:443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: edge-oidc
        optionss: { }
  - name: management
    bindPoints:
      - interface: 127.0.0.1:1282
        address: controller1.domain.com:8443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: fabric
        options: { }
      - binding: zac
        options:
          location: /opt/openziti/share/console
          indexFile: index.html

And if try to go https://ziti.domain.com I will get "not trusted" and when checking the certificate it's ziti's internal certificate and not LE one?

2

You MUST not use the same domain for the alternate certs as the overlay network. If you do, you will experience non deterministic behavior.

I expect you have used the same domain for the controller and router advertised addresses as the let's encrypt cert. That is a bad idea and in practice, won't be successful.

Assuming you have done that, just assign two fqdn or use a wildcard cert to isolate the domain.

3

So, I understand different fqdn, but should the domain part be also different?

I mean now I have

So, should I use?

4

No those are sufficiently different that i wouldn't expect a problem. Somehow the SNI selection is either failing, or the cert the controller is presenting is invalid.

I would check using something like:

openssl s_client -connect ec2-52-201-240-80.compute-1.amazonaws.com:1280 | openssl x509 -text | grep Alternative -A2

If you run that (that's my controller) you'll see the Subject Alternative Names returned:

$ openssl s_client -connect ec2-52-201-240-80.compute-1.amazonaws.com:1280 | openssl x509 -text | grep Alternative -A2
depth=1 CN = ziti-controller-web-intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = ziti-controller-web-identity
verify return:1
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti-controller, DNS:ziti-controller-client, DNS:ziti-controller-client.ziti, DNS:ziti-controller-client.ziti.svc, DNS:ziti-controller-client.ziti.svc.cluster.local, DNS:ec2-52-201-240-80.compute-1.amazonaws.com, DNS:ec2-52-201-240-80.compute-1.amazonaws.com, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1

Let's make sure the certs that are coming back are for the url you think you're connecting to? It's easy to screw up ports etc. I'll try to revise my HAProxy setup and see if I did anything differently. I assume you found that discourse post but if not you can scan Using a Single Port for OpenZiti Components to see if there's any hints in there.

5

Thanks @TheLumberjack to guiding right direction :slight_smile:

To get it working I needed to remove alt_server_certs from identity and move it to below web.

I think this is because I have split config, clients on port 443 and management on port 8443.

identity:
  cert:        "pki/intermediate/certs/client.chain.pem"
  server_cert: "pki/intermediate/certs/server.chain.pem"
  key:         "pki/intermediate/keys/server.key"
  ca:          "pki/root/certs/root.cert"
  # Move LE alt_server_certs from here to web section
...

web:
  - name: client-management
    bindPoints:
      - interface: 127.0.0.1:1280
        address: controller1.domain.com:443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
      # To here
      alt_server_certs:
      - server_cert: "/etc/letsencrypt/live/ziti.domain.com/fullchain.pem "
        server_key:  "/etc/letsencrypt/live/ziti.domain.com/privkey.pem"

    
   options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: edge-oidc
        optionss: { }
...
1 Like