Making ZAC dark

TheLumberjack · June 9, 2022, 1:56am

Exactly. That’s the sort of experience we will get around to delivering!

dmuensterer · July 18, 2023, 11:26am

Following up on this as we are currently checking our setup and how/where we can ensure additional security & availability
We tried to separate the management api from the “edge” API. When doing so, there is a TCP listener behind the specified port, however any connect via e.g. ziti edge login results in an error message:

error: unable to retrieve server certificate authority from https://localhost:8440: pkcs7: input data is empty

Old config, works with the management interface on port 8441:

v: 3
db:                     "/home/ziti/.ziti/quickstart/zt/db/ctrl.db"
identity:
  cert:                 "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-client.cert"
  server_cert:          "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-server.chain.pem"
  key:                  "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/keys/zt-server.key"
  ca:                   "/home/ziti/.ziti/quickstart/zt/pki/cas.pem"
ctrl:
  listener:             tls:0.0.0.0:6262
mgmt:
  listener:             tls:0.0.0.0:10000
healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s
edge:
  api:
    sessionTimeout: 30m
    address: zt.mydomain.com:8441
  enrollment:
    signingCert:
      cert: /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/certs/zt-signing-intermediate.cert
      key:  /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/keys/zt-signing-intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-management
        options: { }

Attempt to separate edge-management:

#### Changes only from here ####
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-management
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:8440
        address: zt.mydomain.com:8440
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.3
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }

I tried to find the issue by using @TheLumberjack example as well as the helmchart examples but couldn’t get it to work.

root@zt:~# netstat -tulpn | grep 8440
tcp6       0      0 :::8440                 :::*                    LISTEN      3081520/ziti-contro

root@zt:~# curl -k -v https://localhost:8440
*   Trying ::1:8440...
* Connected to localhost (::1) port 8440 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; L=Charlotte; O=NetFoundry; OU=ADV-DEV; CN=zt.mydomain.com server certificate
*  start date: Oct 15 22:24:59 2022 GMT
*  expire date: Oct 15 22:25:57 2023 GMT
*  issuer: C=US; L=Charlotte; O=NetFoundry; OU=ADV-DEV; CN=zt.mydomain.com-intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5580e17fd180)
> GET / HTTP/2
> Host: localhost:8440
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404 
< content-length: 0
< date: Tue, 18 Jul 2023 11:25:51 GMT
< 
* Connection #0 to host localhost left intact

Any ideas what’s the issue here?
What exactly does

# the endpoint that management tools connect to the controller over.
mgmt:
  listener:             tls:0.0.0.0:10000

do? I couldn’t really make sense of the comment in the original config. Which kind of management tools are meant here? Does this have anything to do with e.g. ZAC?

TheLumberjack · July 18, 2023, 12:02pm

If you followed the quickstart, you might already have something listening on :8440. I actually just roughly outlined this process the other day over here ZAC On different host than the controller - #2 by TheLumberjack.

Looking at your config there seems to be two issues.

First, in “client-management” you still have “edge-management” in your “apis” section. You’d want to comment/remove that section. I’d also move “fabric” down to the ‘management’ section…

Here’s those changes (and I’ve also moved the port to 18441 vs 8440):

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:18441
        address: zt.mydomain.com:18441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.3
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: fabric
        options: { }

After you apply that config, if you run sudo ss -lntp | grep 8441 you should see two results.

Here’s a short (4 min) video showing me doing that process if it helps:

dmuensterer · July 18, 2023, 9:10pm

Thanks, I’ve made the changes and the controller is now listening on port 18441, however when trying to login I still get an error

root@zt:~# sudo ss -lntp | grep 8441
LISTEN 0      4096       127.0.0.1:18441      0.0.0.0:*    users:(("ziti-controller",pid=3082945,fd=10))
LISTEN 0      4096               *:8441             *:*    users:(("ziti-controller",pid=3082945,fd=11))

root@zt:/home/ziti/.ziti/quickstart/zt/ziti-bin/ziti-v0.26.10# ./ziti edge login localhost:18441/edge/management/v1
error: unable to retrieve server certificate authority from https://localhost:18441: pkcs7: input data is empty

root@zt:/home/ziti/.ziti/quickstart/zt/ziti-bin/ziti-v0.26.10# curl -sk https://localhost:18441/edge/management/v1
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://zt.mydomain.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://zt.mydomain.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://zt.mydomain.com:18441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2022-10-13T15:31:04Z","revision":"72978b5aa932","runtimeVersion":"go1.19.1","version":"v0.26.10"},"meta":{}}

TheLumberjack · July 18, 2023, 9:13pm

~~First, you’re listen on “all interfaces” again. Is that intentional? I would replace the 0.0.0.0 with 127.0.0.1…~~ haha I saw the edit while I was typing…

I see you’re using ziti v0.26.10. I think there might have been a bug before that version around logging in. Could you try with v0.29.0 (the latest) and see what happens? I think that’s the issue here.

dmuensterer · July 18, 2023, 9:14pm

haha I saw the edit while I was typing…

Haha! jep saw that mistake just while pasting it here

dmuensterer · July 18, 2023, 9:19pm

Could you try with v0.29.0 (the latest) and see what happens?

I didn't upgrade the controller yet, is there anything special I should be aware about?
Just stopping the controller, downloading the new ziti binary, changing the systemd services respectively from ziti-controller to ziti controller etc.?
Or is there maybe even a update/upgrade doc I haven't seen yet?

TheLumberjack · July 18, 2023, 9:22pm

You actually don’t need to update the controller nor to bounce it. The issue is with the “client side” of the ziti CLI. So, to be safe, just download the latest ziti and run it specifically. For example:

mkdir /tmp/latest-ziti
cd /tmp/latest-ziti/
wget https://github.com/openziti/ziti/releases/download/v0.29.0/ziti-linux-amd64-0.29.0.tar.gz
tar xvfz ziti-linux-amd64-0.29.0.tar.gz
chmod +x ./ziti # i don't know why but you need to make it executable now
./ziti --version
v0.29.0

just use ./ziti or /tmp/latest-ziti/ziti to login. That make sense?

dmuensterer · July 18, 2023, 9:28pm

Thanks, makes sense, unfortunately still the same error. I'm unsure what the problem ist here...

root@zt:/tmp/latest-ziti# ./ziti --version
v0.29.0
root@zt:/tmp/latest-ziti# ./ziti edge login localhost:18441/edge/management/v1
error: unable to retrieve server certificate authority from https://localhost:18441: pkcs7: input data is empty

root@zt:/tmp/latest-ziti# cat /home/ziti/.ziti/quickstart/zt/zt.yaml # Only searched and replaced our domain for privacy purposes here on the open forum :-)
v: 3
db:                     "/home/ziti/.ziti/quickstart/zt/db/ctrl.db"
identity:
  cert:                 "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-client.cert"
  server_cert:          "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-server.chain.pem"
  key:                  "/home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/keys/zt-server.key"
  ca:                   "/home/ziti/.ziti/quickstart/zt/pki/cas.pem"
ctrl:
  listener:             tls:0.0.0.0:6262
mgmt:
  listener:             tls:0.0.0.0:10000
healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s
edge:
  api:
    sessionTimeout: 30m
    address: zt.mydomain.com:8441
  enrollment:
    signingCert:
      cert: /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/certs/zt-signing-intermediate.cert
      key:  /home/ziti/.ziti/quickstart/zt/pki/zt-signing-intermediate/keys/zt-signing-intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: zt.mydomain.com:8441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
  - name: management
    bindPoints:
      - interface: 127.0.0.1:18441
        address: zt.mydomain.com:18441
    identity:
      ca:          "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-intermediate.cert"
      key:         "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/keys/zt.mydomain.com-server.key"
      server_cert: "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-server.chain.pem"
      cert:        "/home/ziti/.ziti/quickstart/zt/pki/zt.mydomain.com-intermediate/certs/zt.mydomain.com-client.cert"
    options:
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.3
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: fabric
        options: { }

What I find so interesting about this is that it works once I default the management-api back on port 8441 :ziggy_question:

TheLumberjack · July 18, 2023, 9:35pm

Ok. I wonder if this is “an ipv6 thing”… Try 127.0.0.1 not localhost…

TheLumberjack · July 18, 2023, 9:36pm

“localhost” is an oddball because it can either map to ::1 or 127.0.0.1/8 … and we’ve seen ::1 take precedence like this in the past including when offloading traffic to ‘localhost’. This user with minecraft and I had a hard time tracking this one down: Problems with Docker Compose setup and Minecraft Example - #31 by TheLumberjack

dmuensterer · July 19, 2023, 6:39am

Huh, somehow still not possible to connect to it:

root@zt:/tmp/latest-ziti# ./ziti edge login 127.0.0.1:18441/edge/management/v1
error: unable to retrieve server certificate authority from https://127.0.0.1:18441: pkcs7: input data is empty

Should I try upgrading the controller at this point?

TheLumberjack · July 19, 2023, 9:49am

Maybe. I didn’t see a bug around this filled, but it’s possible there was one that we ended up fixing. I suppose it’ll give you a chance to test your upgrade and DR plan a bit though.

If it was me, I works start by snapshotting the db to back it up. Once the db is backed up, you can do one of two things. Simply replace the binary in the current location, or you can you the ziti-cli-functions.sh to get the latest version of ziti, create new systemd files, and restart the services. Note that with v0.29.0, the latest release, you’ll need to fetch the latest ziti-cli-functions.sh and numerous environment variables changed. We tried our best to document how to change to the latest version but it was a pretty big change, there’s a reasonable chance it’s going to end up having a missing value. For this reason, I’d just ask you to backup your .env file the quickstart generated as well until you get through this one upgrade. Technically, you don’t even need this file but it’s convenient to have, because it normalizes the commands I can provide to support you… I’ll even hop on a call with you to with through this if you like, but it should hopefully all be very straightforward…

dmuensterer · July 19, 2023, 1:24pm

Thanks, upgrading worked and so does the login!
Do you recommend keeping the .env file? As far as I understand it’s now inconsistent as the commands changed anyways?

TheLumberjack · July 19, 2023, 1:33pm

Amazing! Glad to hear. Thanks for following up.

Realistically, once you get used to what the config files are "supposed to look like", and you're used to logging in via the CLI or via ZAC, that file has limited usefulness... The best reason to keep it around is just so in the future, if there's any question like 'where is my pki' or whatever, when you ask here on discourse I can refer to the variables for you, making it easier for me to troubleshoot the issue and support people like yourself.

But by all means, you do not need to keep it around. In fact, I image people who've looked in that file have noticed that it will contain the user and pwd to use for auth to the controller. That's totally a convinience for you (and has proven to be a useful thing for new learners), and again once you're familiar with ziti and logging in, etc... I'd probably remove those entries at least, if not the whole file... The quickstart was originally planned to be for "dev/trial/getting used to ziti" type use, but it's proven to be useful and people continue to use it for longer than the original intention because it (usually) just works. We have plans to document the "from quickstart to production" journey, but like all things it just takes time. I'm sure we'll be getting to that relatively soon though.

So that's a long way to say, "it's totally up to you".

TheLumberjack · July 19, 2023, 1:37pm

To tie this off, the commands are all the same, and "most" of the env variables are the same but we recently went through an "environment variable" cleanup (there were too many due to the organic growth of the quickstart) and that cleanup chagned some variables is all. So keep it, ditch it, it probably doesn't much matter now

dmuensterer · July 19, 2023, 1:45pm

Gotcha, thanks! I was referring to the Release Notes of 0.29.0 where it says

All ZITI_EDGE_ROUTER_ variables have been changed to just ZITI_ROUTER_ .

I'll delete it as I don't think there's any value in it anymore.

I've moved the whole quickstart folder in the past and adjusted all references to the path via sed because it was in the /root/ directory and I wanted Ziti to have a dedicated user.
Every reference was correctly updated except for the identity 'default':
Using username: admin from identity 'default' in config file: /root/.config/ziti/ziti-cli.json

I want it to use the file in /home/ziti/.config/ziti/ziti-cli.json. Do you know where I can change this? I can't find a file referencing ziti-cli.json...:

root@zt:/home/ziti/.ziti/quickstart/zt# grep -rnw . -e "ziti-cli.json"
root@zt:/home/ziti/.ziti/quickstart/zt#

TheLumberjack · July 19, 2023, 1:56pm

If you were using the ZITI_HOME environment variable to configure where your ziti CLI profiles were stored, you should now use ZITI_CONFIG_DIR instead.

So set ZITI_CONFIG_DIR

Topic		Replies	Views
How do I use ZAC with quickstart docker-compose? Ziti Administration Console	8	365	March 20, 2023
Dark ZAC with Management API on Localhost 18441 Ziti Administration Console	3	235	August 9, 2022
Docker-compose.yaml with ziti simple start and caddy for reverse proxy https	5	169	August 29, 2023
Issues after going dark + upgrade General Questions	17	269	October 21, 2023
Ziti Administration Console Updates Ziti Administration Console	8	139	December 8, 2023

Making ZAC dark

Related Topics