Host OpenZiti Anywhere Unable to enroll

Support
1

After Host OpenZiti Anywhere configures the controller and router, the status shows running. Use the command to generate the identity. Get the identity jwt and go to Ziti Desktop Edge enroll on MacOS. It prompts CONTROLLER_UNAVAILABLE. The following is the log:

(4329)[2024-03-18T07:49:55.255Z] INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting enrollment at (2024-03-18T07: 49:57.686)
(4329)[2024-03-18T07:49:57.748Z] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[today.icebear.store] request failed: -53(software caused connection abort)
(4329)[2024-03-18T07:49:57.748Z] ERROR ziti-sdk:ziti_enroll.c:221 well_known_certs_cb() /Users/runner/work/ziti-sdk-swift/ziti-sdk-swift/deps/ziti -tunnel-sdk-c/build-macosx-arm64/_deps/ziti-sdk-c-src/library/ziti_enroll.c:139 - ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)
[2024-03-18T07:49:57:749Z] ERROR CZiti:ZitiEnroller.swift:213 on_enroll() CONTROLLER_UNAVAILABLE
[2024-03-18T07:49:57:749Z] ERROR CZiti:Ziti.swift:327 enroll() Optional(Error Domain=ZitiError Code=-7 "CONTROLLER_UNAVAILABLE" UserInfo={NSLocalizedDescription=CONTROLLER_UNAVAILABLE})

jwt iss is https://today.icebear.store:8441, and the domain name resolution IP is 8.141.87.121

ziti-controller log:
{"_context":"tls:0.0.0.0:8441","error":"write tcp 172.23.129.26:8441-\u003e60.16.5.38:45091: write: connection reset by peer","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"60.16.5.38:45091"}

2

ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)

This can happen for a few different reasons. The most common of which are:

  • the jwt issuer url is invalid
  • the certificate presented by the iss url was not signed by the same key that signed the jwt and server cert
  • the controller is offline or unreacable
  • the JWT has expired. it's only valid for 3 hours by default. see the edge.enrollment.edgeIdentity.duration setting of the controller

Check all of these first. If you still have a problem, we'll try to diagnose more

1 Like
3

"iss": "https://today.icebear.store:8441"可用
curl https://ctrl.today.icebear.store:8441/
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-03-13T23:53:41Z","capabilities":[],"revision":"6f295a411d3f","runtimeVersion":"go1.21.8","version":"v0.33.1"},"meta":{}}
curl -sk https://today.icebear.store:8441
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://today.icebear.store:8441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-03-13T23:53:41Z","capabilities":[],"revision":"6f295a411d3f","runtimeVersion":"go1.21.8","version":"v0.33.1"},"meta":{}}

controller running
● ziti-controller.service - Ziti-Controller
Loaded: loaded (/etc/systemd/system/ziti-controller.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-03-18 11:42:43 CST; 21h ago
Main PID: 10852 (ziti)
Tasks: 8 (limit: 1931)
Memory: 92.0M
CPU: 1min 15.678s
CGroup: /system.slice/ziti-controller.service
└─10852 /home/ecs-assist-user/.ziti/quickstart/iZ2ze2dwhfc8ehsqxmy7r1Z/ziti-bin/ziti-v0.33.1/ziti controller run /home/ecs-assist-use…

jwt has not expired

Regarding the second item, I used docker certbot to generate the wildcard certificate *.today.icebear.store, using "source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli -functions.sh)"; expressInstall" generates controllers and routes, and jwt is obtained by the console creation identity.

I don't know what went wrong

1 Like
4

when I use openssl to connect to your port, it fails:

openssl s_client -connect ctrl.today.icebear.store:8441
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 326 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

It looks like no certificate was returned. OpenZiti will maintain it's own PKI. it's really important if you use DNS entries, to make one that isnt' covered by a wildcard cert, and that wildcard certs aren't enabled for the advertised address. I think that's what's going wrong for you here. I think (if i remember) you were doing browzer work before and you're trying to use OpenZiti tunnelers/SDK now?

Would you share your controller config file with me via DM so I can look at it? I expect you have alt_server_certs enabled on it, is that true?

Knowing that you have a wildcard certificate, I feel like that's probably where you're going wrong. I should do a ziti tv on this since it's sort of complex to explain in a discourse post... I think you're in a bit of a complex situation...

Are you still developing things? Can we "start fresh"? If so, I would do this... Since you own your own DNS, you should make a DNS entry that the OpenZiti controller can have a PKI of its own to maintain that is NOT overlapped by a browzer/alternate cert. Let's just call it "overlay.icebear.store". Make a DNS entry for "overlay.icebear.store" set to the wildcard IP (8.141.87.121)...

That "overlay.icebear.store" will be where the router an the controller live (for starters). Follow the browzer example but make sure you set the EXTERNAL_DNS to "overlay.icebear.store". That way it's different than ".today.icebear.store". When the express install completes, you'll have a full PKI for "overlay.icebear.store" that OpenZiti can own and operate -- that's important... And you'll be setup for ".today.icebear.store" as well.

I think I'll do a ziti tv on this topic Friday at 11 AM ET. The Ziti TV will be to cover how to setup an OpenZIti install for use with browzer and this particular pitfall...

Maybe that gives you all the information you need to be successful? :confused: I feel like we need a very detailed guide for this but that will take some time...

5

Yes, I plan to use the tunneller part, the process of creating controller routing using browser, the certificate part of my controller configuration:

identity:
cert: "/home/ecs-assist-user/.ziti/quickstart/iZ2ze2dwhfc8ehsqxmy7r1Z/pki/iZ2ze2dwhfc8ehsqxmy7r1Z-intermediate/certs/iZ2ze2dwhfc8ehsqxmy7r1Z-client.cert"
server_cert: "/home/ecs-assist-user/.ziti/quickstart/iZ2ze2dwhfc8ehsqxmy7r1Z/pki/iZ2ze2dwhfc8ehsqxmy7r1Z-intermediate/certs/iZ2ze2dwhfc8ehsqxmy7r1Z-server.chain.pem"
key: "/home/ecs-assist-user/.ziti/quickstart/iZ2ze2dwhfc8ehsqxmy7r1Z/pki/iZ2ze2dwhfc8ehsqxmy7r1Z-intermediate/keys/iZ2ze2dwhfc8ehsqxmy7r1Z-server.key"
ca: "/home/ecs-assist-user/.ziti/quickstart/iZ2ze2dwhfc8ehsqxmy7r1Z/pki/cas.pem"
alt_server_certs:
- server_cert: "/etc/letsencrypt/live/today.icebear.store/fullchain.pem"
server_key: "/etc/letsencrypt/live/today.icebear.store/privkey.pem"

I can start over. Next, I don't use the wildcard certificate, and directly use "overlay.icebear.store" as EXTERNAL_DNS. Ziti will automatically generate the corresponding certificate. I only need to resolve the DNS overlay.icebear.store to my ip, like this process?

6

Yes, I think so assuming I understand what you posted. At the end of the day, when you use openssl, it needs to return a certificate valid for the URL you're connecting to. The one above is not returning a certificate at all which is a problem.

8

I successfully completed the tunnel process using Local - No Docker | OpenZiti, but use the https://openziti.io/docs/learn/quickstarts/network/hosted， overlay.icebear.store still does not work.

openssl s_client -connect overlay.icebear.store:8441
CONNECTED(00000005)
write:errno=54

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 323 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

I established a mapping between overlay.icebear.store and 8.141.87.121 in DNS. It feels no different from the local method, except that the environment variables and ports have changed.

9

I must admit, I have never seen the HTTP port NOT return a certificate like this. What version are you running? Is there any kind of WAF in front of this? That's really strange to me.

On a whim, i asked for a different server name and get a certificate back but this is very unexpected imo...

openssl s_client -connect overlay.icebear.store:8441 -servername clear.this.is.wrong.com </dev/null

and then it DOES return a cert:

$ openssl s_client -connect overlay.icebear.store:8441 -servername clear.this.is.wrong.com </dev/null
CONNECTED(00000003)
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-root-ca Root CA
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-root-ca Root CA
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-intermediate
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = overlay.icebear.store server certificate
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = overlay.icebear.store server certificate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 21 06:27:41 2024 GMT; NotAfter: Mar 21 06:28:40 2025 GMT
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-root-ca Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 21 06:27:37 2024 GMT; NotAfter: Mar 19 06:28:35 2034 GMT
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-root-ca Root CA
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = iZ2ze2dwhfc8ehsqxmy7r1Z-edge-controller-root-ca Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 21 06:27:32 2024 GMT; NotAfter: Mar 19 06:28:31 2034 GMT

Then when I use the 'correct' DNS entry, no certs come back.Curl also doesn't work:

$ curl https://overlay.icebear.store:8441
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to overlay.icebear.store:8441

This behavior is definitely not expected. I've never seen this particular situation. You're following the expressInstall exactly as documented? Did you make any changes to those steps?

This is all very odd...

10

I have tried many times, using exactly the Host OpenZiti Anywhere | OpenZiti process, whether using the ziti pki automatically generated certificate, or using the overlay.icebear.store certificate generated by certbot, both work Unable to access and the client prompts for enroll:
ziti-sdk-c-src/library/ziti_enroll.c:139 - ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)
[2024-03-22T01:17:14:207Z] ERROR CZiti:ZitiEnroller.swift:213 on_enroll() CONTROLLER_UNAVAILABLE
[2024-03-22T01:17:14:207Z] ERROR CZiti:Ziti.swift:327 enroll() Optional(Error Domain=ZitiError Code=-7 "CONTROLLER_UNAVAILABLE" UserInfo={NSLocalizedDescription=CONTROLLER_UNAVAILABLE})

Use your own computer terminal to run the command (I now change it to test.icebear.store):
gin:~ gin$ openssl s_client -connect test.icebear.store:8441
CONNECTED(00000005)
write:errno=54

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 320 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

gin:~ gin$ curl https://test.icebear.store:8441
curl: (35) Recv failure: Connection reset by peer

11

I got on a call with @McGonagall666 last night and we did some troubleshooting. it turns out SNI-based TLS connections were being reset immediately for reasons I don't understand. If anyone else comes across this discourse post, make sure you can use opessl to connect to your controller/router/etc and ensure you see certificates returned.

To diagnose - use the IP of your host:

openssl s_client -connect 3.18.113.172:8441 </dev/null

The result must be nearly identical to what you see when you use the DNS:

openssl s_client -connect ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441 </dev/null

When I tried this, the SSL handshake changes a tiny bit:

image
image1052×238 17.3 KB