iOS unable to connect after edge router certificate rolled

Support General Questions
1

So, I updated my iPhone to 18.5 yesterday. After restart Ziti client on the phone is not working. I checked with my laptop and that was working as expected. I started nosing around, and found that it appears that the edge-router-cert also updated same night. However, all client devices are online EXCEPT my iPhone.
What I have done:

  • Restarted iPhone
  • Toggled Ziti on/off
  • Re-enrolled identity on iPhone
  • Added a new identity onto iPhone
  • Removed app from iPhone deleting data and reinstalling

App version on iPhone is 2.47
on ZAC I get
image

On the edge-router logs I get

[ 854.699]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51298] error=[local error: tls: bad record MAC]} handshake failed
[ 888.877]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51303] error=[local error: tls: bad record MAC]} handshake failed
[ 910.700]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51307] error=[local error: tls: bad record MAC]} handshake failed
[ 970.800]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51314] error=[local error: tls: bad record MAC]} handshake failed
[ 976.966]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51315] error=[local error: tls: bad record MAC]} handshake failed
[1077.848]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51326] error=[local error: tls: bad record MAC]} handshake failed
[1224.394]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[local error: tls: bad record MAC] remote=[192.168.9.28:51331]} handshake failed
[1231.785]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[192.168.9.28:51333] error=[local error: tls: bad record MAC]} handshake failed

Which has been proven before to be expired certificate related. However all other devices are happily connecting to that edge router

I see the ziti edge controller certificate is expired, and has been for two years however this has only failed in the last couple of days. I will work on getting this updated.

Currently running ziti 1.1.10. This is the first time certs have updated since upgrading from OpenZiti 0.38 (ish) -> 1.1.10.

2

So I have done a bit more to try and resolve this. Seems I have been here before (well nearly): iOS Ziti Client cannot connect to controller. In that instance, I had a red dot, but now I have a green dot, and an API connection.

So, I decided to recheck my PKI and make sure it is all correct. Especially since the router changed the certificate. I had not updated the client certificate on the controller for the last couple of years. Mostly because it worked. But, could this be the issue? no :frowning:

So, going back to basics, my cas.pem still contained some intermediate certs. I had thought that I had removed them. I followed through this: Python SDK throws "Controller not available" error - #27 by dmuensterer and got that sorted.

I then followed Certificate Expired - #2 by gooseleggs to recreate all certificates, including client certs.

Now, following this: PKI Troubleshooting | OpenZiti from top to bottom I am getting ============ SUCCESS! ============ for all tests. I cannot test the iPhone identity, but I used an identity that is/was working.

I have then re-enrolled the iOS phone, and still no dice.

So, I have now verified (and likely corrected) the PKI and all tests are returning good, so where I thought that PKI might have been a problem, I don't see that it is now.

3

OK - I have it working. I haven't got a fix, but I have it working (workaround). So, carrying on from above, Windows clients were able to connect and access services. iPhone was still not able to do it.

I then started to use openssl s_client -connect zitiedgerouter.thesmithcave.nz:3022. I was getting this:

 openssl s_client -connect zitiedgerouter.thesmithcave.nz:3022 -showcerts
CONNECTED(00000003)
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = KpnBJA6HTS
verify return:1
---
Certificate chain
 0 s:C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = KpnBJA6HTS
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 16 05:28:16 2025 GMT; NotAfter: May 16 05:29:16 2026 GMT

Upon closer inspection it turns out that the certificate is missing a second intermediate certificate. Once I clicked to that, and added it to the edge client and server certificates, it came into life.

[kelvins@Server2 openziti]$ openssl s_client -connect zitiedgerouter.thesmithcave.nz:3022
CONNECTED(00000003)
depth=3 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-root-ca Root CA
verify return:1
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate_spurious_intermediate
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
verify return:1
depth=0 C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = KpnBJA6HTS
verify return:1
---
Certificate chain
 0 s:C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = KpnBJA6HTS
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 16 05:28:16 2025 GMT; NotAfter: May 16 05:29:16 2026 GMT
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate_spurious_intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun  5 20:42:16 2022 GMT; NotAfter: Jun  2 20:43:15 2032 GMT
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate_spurious_intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-root-ca Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun  5 20:42:15 2022 GMT; NotAfter: Jun  2 20:43:14 2032 GMT

So, the question is why the router certificates are not adding all the intermediate certificates when the certificate was renewed?
The controller uses a different CA for router certs - not sure if this normal, or a hangover from pre 0.28 versions of the PKI. I created a new router and used a jwt to initialize the router config. That was also missing the secondary certificate.

Thoughts?

4

I am not sure how to change the title of the post. It should read iOS unable to connect after edge router certificate rolled or something like that.

5

Today with the fabric working across all devices, I created a test router. The certificate only contained the first intermediate certificate and not both intermediate certificates. This would/could likely be a bug?

6

Hello again @gooseleggs. Thanks for sharing the problem with all of the details!

There have been a couple of issues related to preserving certificate chains on enrollments and re-enrollments. Some of the issues only affected the apple clients and others affected the controller.

The apple client issues would 1) cause the client to only save the first certificate in the chain of identity certs when enrolling a new identity, and 2) prevent the client from saving any updates to identity certificates when they are extended. These issues are fixed ion the upcoming (I suspect this week or next) release of the apple tunnelers. DM me an Apple ID email address if you'd like to try the pre-release builds from test flight before they hit the apple stores.

The controller issue prevented the intermediates from being included in certificate extensions. Based on your current state I suspect this is what you are seeing with your edge router.