Making ZAC dark

Following up on this as we are currently checking our setup and how/where we can ensure additional security & availability :slight_smile:
We tried to separate the management api from the “edge” API. When doing so, there is a TCP listener behind the specified port, however any connect via e.g. ziti edge login results in an error message:

error: unable to retrieve server certificate authority from https://localhost:8440: pkcs7: input data is empty

Old config, works with the management interface on port 8441:

v: 3
db:                     "/home/ziti/.ziti/quickstart/zt/db/ctrl.db"
identity:
  cert:                 "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-client.cert"
  server_cert:          "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-server.chain.pem"
  key:                  "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/keys/zt-server.key"
  ca:                   "/home/ziti/.ziti/quickstart/zt/pki/cas.pem"
ctrl:
  listener:             tls:0.0.0.0:6262
mgmt:
  listener:             tls:0.0.0.0:10000
healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s
edge:
  api:
    sessionTimeout: 30m
    address: zt.mydomain.com:8441
  enrollment:
    signingCert:
      cert: /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/certs/zt-signing-intermediate.cert
      key:  /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/keys/zt-signing-intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-management
        options: { }

Attempt to separate edge-management:

#### Changes only from here ####
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-management
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:8440
        address: zt.mydomain.com:8440
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.3
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }

I tried to find the issue by using @TheLumberjack example as well as the helmchart examples but couldn’t get it to work.

root@zt:~# netstat -tulpn | grep 8440
tcp6       0      0 :::8440                 :::*                    LISTEN      3081520/ziti-contro 
root@zt:~# curl -k -v https://localhost:8440
*   Trying ::1:8440...
* Connected to localhost (::1) port 8440 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; L=Charlotte; O=NetFoundry; OU=ADV-DEV; CN=zt.mydomain.com server certificate
*  start date: Oct 15 22:24:59 2022 GMT
*  expire date: Oct 15 22:25:57 2023 GMT
*  issuer: C=US; L=Charlotte; O=NetFoundry; OU=ADV-DEV; CN=zt.mydomain.com-intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5580e17fd180)
> GET / HTTP/2
> Host: localhost:8440
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404 
< content-length: 0
< date: Tue, 18 Jul 2023 11:25:51 GMT
< 
* Connection #0 to host localhost left intact

Any ideas what’s the issue here?
What exactly does

# the endpoint that management tools connect to the controller over.
mgmt:
  listener:             tls:0.0.0.0:10000

do? I couldn’t really make sense of the comment in the original config. Which kind of management tools are meant here? Does this have anything to do with e.g. ZAC?