From scratch script and HA

TheLumberjack · February 27, 2025, 6:30pm

1.4.1 pre-release was created. it should fix the issue. note the ziti create config command changed. gone is -minCluster and --clustered has entered. same for ziti agent controller -> ziti agent cluster (if you use 1.4+)

voev · March 10, 2025, 3:16am

Hi! Great project! I managed to set up a cluster of 3 nodes using the scripts from this thread. I used ziti version 1.4.3 and the --clustered flag to create the configurations. However, I can't connect to the controllers using Ziti Desktop Edge for macOS and Android. The enrollment process completes, and the token is consumed. DNS records resolve correctly, and the required DNS and IP are present in the certificates.

voev · March 10, 2025, 3:56am

in controllers console, 10.1.10.74 - pc with Ziti Desktop Edge

[2278.271]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64940] error=[remote error: tls: internal error]} handshake failed
[2283.297]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64941] error=[remote error: tls: internal error]} handshake failed
[2288.319]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64942] error=[remote error: tls: internal error]} handshake failed
[2293.348]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64943] error=[remote error: tls: internal error]} handshake failed
[2298.369]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64944] error=[remote error: tls: internal error]} handshake failed
[2303.398]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64945]} handshake failed
[2308.428]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64946] error=[remote error: tls: internal error]} handshake failed
[2313.456]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64947]} handshake failed
[2318.486]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64948] error=[remote error: tls: internal error]} handshake failed
[2323.515]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64949] error=[remote error: tls: internal error]} handshake failed
[2328.535]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64950]} handshake failed
[2333.561]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64951] error=[remote error: tls: internal error]} handshake failed
[2338.595]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64952] error=[remote error: tls: internal error]} handshake failed
[2343.632]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64953] error=[remote error: tls: internal error]} handshake failed
[2348.655]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64954] error=[remote error: tls: internal error]} handshake failed
[2353.675]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64955] error=[remote error: tls: internal error]} handshake failed
[2358.718]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64956] error=[remote error: tls: internal error]} handshake failed
[2363.755]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64957] error=[remote error: tls: internal error]} handshake failed
root@ec ziti:#[2369.412]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64958]} handshake failed
root@ec ziti:#[2373.808]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64959]} handshake failed
[2378.835]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64960] error=[remote error: tls: internal error]} handshake failed
[2383.882]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64961] error=[remote error: tls: internal error]} handshake failed
[2388.897]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64962] error=[remote error: tls: internal error]} handshake failed
[2393.929]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64963] error=[remote error: tls: internal error]} handshake failed
[2398.977]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {error=[remote error: tls: internal error] remote=[10.1.10.74:64964]} handshake failed
[2404.003]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:443]: {remote=[10.1.10.74:64965] error=[remote error: tls: internal error]} handshake failed

dig ec.nsk.z

; <<>> DiG 9.10.6 <<>> ec.nsk.z
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26883
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ec.nsk.z.			IN	A

;; ANSWER SECTION:
ec.nsk.z.		0	IN	A	10.1.10.201

;; Query time: 3 msec
;; SERVER: 10.1.10.53#53(10.1.10.53)
;; WHEN: Mon Mar 10 11:00:33 +07 2025
;; MSG SIZE  rcvd: 53

openssl s_client -connect ec.nsk.z:443 < /dev/null | openssl x509 -text |grep Alter -A2
Connecting to 10.1.10.201
depth=2 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=my.root.ca
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=my.root.ca
verify return:1
depth=1 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ec.nsk.z-edge
verify return:1
depth=0 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ec.nsk.z
verify return:1
DONE
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ec.nsk.z, IP Address:127.0.0.1, IP Address:10.1.10.201, URI:spiffe://trustr/controller/ec.nsk.z
    Signature Algorithm: sha256WithRSAEncryption

smilindave26 · March 10, 2025, 11:58am

Hi @voev - welcome to the community! There is an issue with the Ziti Desktop Edge for Mac in the way it stores it's certificate following enrollment that can cause exactly what you're seeing. We're working on it and expect to have this fixed in the next release.

scareything · March 10, 2025, 2:18pm

Which version of Ziti Desktop Edge for macOS are you running? Can you share the appex log?

Thanks.

voev · March 11, 2025, 4:52am

Ziti Desktop Edge for Mac Version 2.47 (525). Where i can find appex log? I have Packet Tunnel and Application logs in Logging menu. I also tried the tunneler for Windows version: 2.5.5.0, it also added an identity but doesn't work.

TheLumberjack · March 11, 2025, 9:51am

You can access the logs from the menubar under Logging/Packet Tunnel... . You can also find the full files in ~/Library/Group\ Containers/MN5S649TXM.ZitiPacketTunnel.group/logs/

Cheers

voev · March 11, 2025, 1:59pm

[2025-03-11T13:56:33:721Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet
[2025-03-11T13:56:33:734Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:33:734Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
[2025-03-11T13:56:33:897Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:227 stopTunnel() 
[2025-03-11T13:56:33:897Z]    INFO CZiti:Ziti.swift:257 executeRunloop() runZiti - loop exited with status 0
[2025-03-11T13:56:33:899Z]    INFO CZiti:ZitiTunnel.swift:278 shutdownZiti() Ziti shutdown complete, status=success
[2025-03-11T13:56:33:899Z]    INFO PacketTunnelProvider:UserNotifications.swift:100 post() Attempting to post Info notification, subitile:Optional("Disconnected"), body:nil, zid:nil
[2025-03-11T13:56:33:905Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:244 stopTunnel() Exiting
[2025-03-11T13:56:34:835Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:38 init() io.netfoundry.ZitiPacketTunnel.PacketTunnelProvider Version: 2.47 (525), OS: Версия 15.3.1 (Выпуск 24D70); ziti-sdk-c version 1.1.5-g2120296(Oct 23 2024 20:07:22)
[2025-03-11T13:56:34:846Z]    INFO PacketTunnelProvider:UserNotifications.swift:94 requestAuth() Auth request authorized? true
[2025-03-11T13:56:34:851Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:58 startTunnelAsync() 
[2025-03-11T13:56:34:851Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:59 startTunnelAsync() options=nil
[2025-03-11T13:56:34:851Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:164 loadConfig() ProviderConfig <PacketTunnelProvider.ProviderConfig: 0x13e115620>
ipAddress: 100.64.0.1
subnetMask: 255.192.0.0
mtu: 4000
dns: 100.64.0.2
fallbackDnsEnabled: false
fallbackDns: 1.1.1.1
interceptMatchedDns: true
lowPowerMode: false
logLevel: 3
logRotateDaily: true
logRotateCount: 5
logRotateSizeMB: 50
[2025-03-11T13:56:34:851Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:87 startTunnelAsync() Setting log level to INFO
[2025-03-11T13:56:34:852Z]    INFO PacketTunnelProvider:Logger.swift:242 updateRotateSettings() Updating log rotate config to daily:true, count:5, sizeMB:50
[2025-03-11T13:56:34:865Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:34:869Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:208 updateTunnelNetworkSettings() route: 100.64.0.1 / 255.192.0.0
[2025-03-11T13:56:34:869Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet
[2025-03-11T13:56:34:880Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:34:880Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
[2025-03-11T13:56:35:021Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet 
     28: name:utun5, type:other
[2025-03-11T13:56:35:034Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:35:034Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
[2025-03-11T13:56:42:826Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet
[2025-03-11T13:56:42:837Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:42:837Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
[2025-03-11T13:56:43:030Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:227 stopTunnel() 
[2025-03-11T13:56:43:030Z]    INFO CZiti:Ziti.swift:257 executeRunloop() runZiti - loop exited with status 0
[2025-03-11T13:56:43:031Z]    INFO CZiti:ZitiTunnel.swift:278 shutdownZiti() Ziti shutdown complete, status=success
[2025-03-11T13:56:43:032Z]    INFO PacketTunnelProvider:UserNotifications.swift:100 post() Attempting to post Info notification, subitile:Optional("Disconnected"), body:nil, zid:nil
[2025-03-11T13:56:43:036Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:244 stopTunnel() Exiting
[2025-03-11T13:56:43:126Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:38 init() io.netfoundry.ZitiPacketTunnel.PacketTunnelProvider Version: 2.47 (525), OS: Версия 15.3.1 (Выпуск 24D70); ziti-sdk-c version 1.1.5-g2120296(Oct 23 2024 20:07:22)
[2025-03-11T13:56:43:134Z]    INFO PacketTunnelProvider:UserNotifications.swift:94 requestAuth() Auth request authorized? true
[2025-03-11T13:56:43:138Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:58 startTunnelAsync() 
[2025-03-11T13:56:43:139Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:59 startTunnelAsync() options=nil
[2025-03-11T13:56:43:139Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:164 loadConfig() ProviderConfig <PacketTunnelProvider.ProviderConfig: 0x1232165f0>
ipAddress: 100.64.0.1
subnetMask: 255.192.0.0
mtu: 4000
dns: 100.64.0.2
fallbackDnsEnabled: false
fallbackDns: 1.1.1.1
interceptMatchedDns: true
lowPowerMode: false
logLevel: 3
logRotateDaily: true
logRotateCount: 5
logRotateSizeMB: 50
[2025-03-11T13:56:43:139Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:87 startTunnelAsync() Setting log level to INFO
[2025-03-11T13:56:43:139Z]    INFO PacketTunnelProvider:Logger.swift:242 updateRotateSettings() Updating log rotate config to daily:true, count:5, sizeMB:50
[2025-03-11T13:56:43:147Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:43:154Z]    INFO CZiti:ZitiTunnel.swift:208 loadAndRunZiti() Starting 6vgVoWPj4J:"Optional("mac.jwt")" at https://ec.nsk.z:443
(47522)[2025-03-11T13:56:43.214Z]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(47522)[2025-03-11T13:56:43.214Z]    INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 1.1.5 @g2120296(HEAD) starting at (2025-03-11T13:56:43.214)
(47522)[2025-03-11T13:56:43.226Z]    INFO ziti-sdk:ziti.c:438 ziti_start_internal() ztx[0] using tlsuv[v0.32.6/OpenSSL 3.3.1 4 Jun 2024]
(47522)[2025-03-11T13:56:43.226Z]    INFO ziti-sdk:ziti_ctrl.c:593 ziti_ctrl_init() ctrl[(null):] using https://ec.nsk.z:443
(47522)[2025-03-11T13:56:43.226Z]    INFO ziti-sdk:ziti.c:507 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ec.nsk.z:443]
(47522)[2025-03-11T13:56:43.257Z]    INFO ziti-sdk:ziti.c:1761 version_pre_auth_cb() ztx[0] connected to HA controller https://ec.nsk.z:443 version v1.4.3(de60092629f9 2025-03-04T16:52:50Z)
[2025-03-11T13:56:48:218Z]    WARN CZiti:ZitiTunnel.swift:232 loadAndRunZiti() Timed out waiting for zidToLoad == 0 (1 of 1 identities have not returned any services
[2025-03-11T13:56:48:219Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:208 updateTunnelNetworkSettings() route: 100.64.0.1 / 255.192.0.0
[2025-03-11T13:56:48:219Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet
[2025-03-11T13:56:48:236Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:48:236Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
(47522)[2025-03-11T13:56:48.235Z]    INFO tunnel-cbs:ziti_dns.c:273 ziti_dns_set_upstream() DNS upstream[1] is set to 10.1.10.53:53
[2025-03-11T13:56:48:354Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:381 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:false, Cellular:false, DNS:true
   Interfaces: 
     20: name:en1, type:wifi 
     18: name:feth758, type:wiredEthernet 
     16: name:feth1902, type:wiredEthernet 
     28: name:utun5, type:other
[2025-03-11T13:56:48:364Z]    WARN PacketTunnelProvider:PacketTunnelProvider.swift:327 getUpstreamDns() No fallback DNS configured. Setting to first resolver: 10.1.10.53
[2025-03-11T13:56:48:364Z]    INFO PacketTunnelProvider:PacketTunnelProvider.swift:366 startNetworkMonitor() Setting fallback DNS to 10.1.10.53
(47522)[2025-03-11T13:56:48.364Z]    INFO tunnel-cbs:ziti_dns.c:273 ziti_dns_set_upstream() DNS upstream[1] is set to 10.1.10.53:53

scareything · March 11, 2025, 4:15pm

Thanks for getting the logs. I see the same thing that you're reporting if I run the version from the App Store (2.47) and load an identity that doesn't have any services assigned to it. Granted the UI status should show as connected even if there aren't any services, so this is a bug.

Have you created services for this identity? If not then go ahead and take the plunge. Hopefully that will get you going. If you do have services assigned to this identity then something else is going on and I'd like to understand it. If you could start a new thread for that I'd appreciate it.

The good news is that the next version that will be going to the App Store soon does not have this issue. Coming soon!

Topic		Replies	Views
HA Controller not adding to cluster	12	55	April 16, 2025
HA implementation questions	4	61	February 25, 2025
ZAC (manage certificate authorities)	8	102	August 27, 2024
Upgrade from standalone to HA setup General Questions	10	106	August 16, 2024
Problem initializing HA cluster Support	14	67	February 4, 2025

From scratch script and HA

Related topics