Ziti v1.1.15 fails initial login

Support
1

I am seeing an issue with ziti v1.1.15.
After initializing and starting the controller When I attempt to login for the first time it fails with:

$ ./bin/ziti edge login ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444 -u "breakglass" -p "breakglass"
Untrusted certificate authority retrieved from server
Verified that server supplied certificates are trusted by server
Server supplied 2 certificates
Trust server provided certificate authority [Y/N]: Y
Server certificate chain written to /home/opc/.config/ziti/certs/ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com
error: no session token returned from login request to https://ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444. Received: {"data":{"checks":[{"details":null,"healthy":true,"id":"bolt.read","lastCheckDuration":"15.12µs","lastCheckTime":"2024-11-06T01:02:17Z"}],"healthy":true},"meta":{}}

The setup works perfectly with v1.1.11 (same controller YAML)

2

The controller logs don't show any error:

{"file":"github.com/openziti/ziti/controller/network/network.go:910","func":"github.com/openziti/ziti/controller/network.(*Network).Run","level":"info","msg":"started","time":"2024-11-06T00:58:47.388Z"}
{"file":"github.com/openziti/xweb/v2@v2.1.3/server.go:197","func":"github.com/openziti/xweb/v2.(*Server).Start","level":"info","msg":"starting ApiConfig to listen and serve tls on 192.168.150.10:8441 for server edge-client-api with APIs: [edge-client]","time":"2024-11-06T00:58:47.388Z"}
{"file":"github.com/openziti/xweb/v2@v2.1.3/server.go:197","func":"github.com/openziti/xweb/v2.(*Server).Start","level":"info","msg":"starting ApiConfig to listen and serve tls on 192.168.170.10:8444 for server edge-mgmt-api with APIs: [fabric edge-management health-checks]","time":"2024-11-06T00:58:47.388Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53896,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:300","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = [h2 http/1.1]","time":"2024-11-06T01:02:25.557Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53896,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2024-11-06T01:02:25.557Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53896,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[h2]","time":"2024-11-06T01:02:25.557Z"}
{"_context":"tls:192.168.170.10:8444","error":"remote error: tls: bad certificate","file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"192.168.170.50:53896","time":"2024-11-06T01:02:25.604Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53904,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:300","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = [h2 http/1.1]","time":"2024-11-06T01:02:25.605Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53904,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2024-11-06T01:02:25.605Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53904,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[h2]","time":"2024-11-06T01:02:25.605Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53904,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:263","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"debug","msg":"selected protocol = 'http/1.1'","remote":"192.168.170.50:53904","time":"2024-11-06T01:02:25.635Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53918,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:300","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = [h2 http/1.1]","time":"2024-11-06T01:02:25.638Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53918,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2024-11-06T01:02:25.638Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53918,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[h2]","time":"2024-11-06T01:02:25.638Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53918,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:263","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"debug","msg":"selected protocol = 'http/1.1'","remote":"192.168.170.50:53918","time":"2024-11-06T01:02:25.668Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53922,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:300","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = []","time":"2024-11-06T01:02:28.343Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53922,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2024-11-06T01:02:28.343Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53922,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:263","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"debug","msg":"selected protocol = ''","remote":"192.168.170.50:53922","time":"2024-11-06T01:02:28.375Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53924,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:300","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"client requesting protocols = []","time":"2024-11-06T01:02:28.377Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53924,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:323","func":"github.com/openziti/transport/v2/tls.(*sharedListener).getConfig","level":"debug","msg":"found handler for proto[]","time":"2024-11-06T01:02:28.377Z"}
{"_context":"tls:192.168.170.10:8444","client":{"IP":"192.168.170.50","Port":53924,"Zone":""},"file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:263","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"debug","msg":"selected protocol = ''","remote":"192.168.170.50:53924","time":"2024-11-06T01:02:28.411Z"}
3

The controller yaml is

$ cat etc/ebea685ad01d442a8515a4f1329d2b8d_config.yaml
v: 3

db: db/ctrl.db

identity:
  cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-client-20240806152549.cert
  server_cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-server-8440-20240806152549.chain.pem
  key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-server-8440-20240806152549.key
  ca: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.chain.pem

ctrl:
  listener: tls:ebea685ad01d442a8515a4f1329d2b8d.priv.zitipoc.com:8440

edge:
  api:
    address: ebea685ad01d442a8515a4f1329d2b8d.pub.zitipoc.com:8441
  enrollment:
    signingCert:
      cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.cert
      key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.key
    edgeIdentity:
      duration: 60m
    edgeRouter:
      duration: 60m

web:
  - name: edge-client-api
    bindPoints:
      - interface: 192.168.150.10:8441
        address: ebea685ad01d442a8515a4f1329d2b8d.pub.zitipoc.com:8441
    identity:
      cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-client-20240806152549.cert
      server_cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-server-8441-20240806152549.chain.pem
      key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-server-8441-20240806152549.key
      ca: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.chain.pem
    apis:
      - binding: edge-client

  - name: edge-mgmt-api
    bindPoints:
      - interface: 192.168.170.10:8444
        address: ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444
    identity:
      cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-client-20240806152549.cert
      server_cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.chain.pem
      key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.key
      ca: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.chain.pem
    apis:
      - binding: fabric
      - binding: edge-management
      - binding: health-checks
4

Hi @plakdawa, thanks for the report. Just so I'm clear, when you test, are you reinstalling everything or are you restoring the 1.1.11 database? I'm just missing how you go in between each version.

5

Hi @TheLumberjack ,
I delete the DB and re-run init and start each time.
So its a brand new setup with empty DB with either version.

6

The error output from the login command unexpectedly contains a health check response, so the login request must've been handled by the wrong binding for some reason.

Will you confirm this by temporarily commenting out the health-checks binding on the mgmt listener, please?

Another variable is the login address. I believe the login command will normally append the mgmt API endpoint, but you can try it with the following in case it works around the issue. It will shed light on what's happening.

Substitute login address: ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444/edge/management/v1

7

Hi @qrkourier
Commenting out health-check in controller yaml works:

$ tail etc/ebea685ad01d442a8515a4f1329d2b8d_config.yaml
        address: ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444
    identity:
      cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-client-20240806152549.cert
      server_cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.chain.pem
      key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.key
      ca: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.chain.pem
    apis:
      - binding: fabric
      - binding: edge-management
#      - binding: health-checks


$ ./bin/ziti edge login ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444 -u "breakglass" -p "breakglass"
Token: 0e023d4d-043b-4d8e-9784-9acaebf9d168
Saving identity 'default' to /home/opc/.config/ziti/ziti-cli.json
8

Restored the health-check and trying with full endpoint also works:

$ tail etc/ebea685ad01d442a8515a4f1329d2b8d_config.yaml
        address: ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444
    identity:
      cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-client-20240806152549.cert
      server_cert: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.chain.pem
      key: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/keys/ebea685ad01d442a8515a4f1329d2b8d-server-8444-20240806152549.key
      ca: pki/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549/certs/ebea685ad01d442a8515a4f1329d2b8d-signing-ca-20240806152549.chain.pem
    apis:
      - binding: fabric
      - binding: edge-management
      - binding: health-checks

$ ./bin/ziti edge login ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444/edge/management/v1 -u "breakglass" -p "breakglass"
Token: 1289f77a-e6ff-4035-ab8b-20441fb9ea6d
Saving identity 'default' to /home/opc/.config/ziti/ziti-cli.json

But short endpoint fails:

$ ./bin/ziti edge login ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444 -u "breakglass" -p "breakglass"
error: no session token returned from login request to https://ebea685ad01d442a8515a4f1329d2b8d.mgmt.zitipoc.com:8444. Received: {"data":{"checks":[{"details":null,"healthy":true,"id":"bolt.read","lastCheckDuration":"10.52µs","lastCheckTime":"2024-11-06T01:38:21Z"}],"healthy":true},"meta":{}}
1 Like
9

Thanks for finding this! We'll track the fix in this GH issue. CLI - login command fails when health-checks enabled · Issue #2523 · openziti/ziti · GitHub

10

Thanks a lot @qrkourier