Segfault of MacOS ZET triggered by no connection traffic

Tunneler Apps Ziti Edge Tunnel
1

Hello. I’m using the ziti-edge-tunneler on MacOS ventura x86_64 instead of Ziti Desktop Edge to get system daemon functionality. Just want to report what might be a bug or an issue running on Darwin.

While working with it, I had at least two TCP sessions open on different ports:

  1. for SSH with periodic keepalives
  2. With remote desktop software

On my local system, the remote desktop software hangs occasionally (separate issue, completely unrelated to Ziti). This remote desktop software hung today while I was working in SSH and it crashed the remote ZET system daemon.

It looks like remote MacOS ZET detected no traffic on one of the channels it had open, which must have been the remote desktop software that had hung since I was active in SSH, and closed the channel, then immediately threw a segmentation fault and exited.

Relevant logs on the MacOS side which had the segmentation fault are:

(298)[   308445.625]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:22]: incoming connection
(298)[   308535.245]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:22]: incoming connection
(298)[   308564.598]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:22]: incoming connection
(298)[   309794.367]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   311534.744]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   313275.114]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   315015.491]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   316755.861]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   318496.236]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   320236.608]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   321976.980]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   323717.346]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   324305.085]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:5900]: incoming connection
(298)[   324620.711]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:5900]: incoming connection
(298)[   324635.111]    INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[$SERVICE], client[$CLIENT] dst_addr[tcp:$MAC_MACHINE:5900]: incoming connection
(298)[   325457.714]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   327198.088]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(298)[   327273.100]   ERROR ziti-sdk:channel.c:563 latency_timeout() ch[0] no read traffic on channel since before latency probe was sent, closing channel
(298)[   327273.100]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:804 on_ziti_event() ztx[$MAC_MACHINE] router $ROUTER_FQDN@tls://$ROUTER_FQDN:$PORT disconnected
(298)[   327273.100]   ERROR ziti-sdk:channel.c:323 on_channel_send() ch[0] write failed [-14/bad address in system call argument]
(298)[   327273.100]    INFO ziti-sdk:channel.c:711 reconnect_channel() ch[0] reconnecting in 6009 ms (attempt = 1)
/bin/sh: line 1:   298 Segmentation fault: 11  ./ziti-edge-tunnel run -I identity

Relevant logs on the Linux side which made the initiating dial connections:

Feb 21 14:09:49 nixos-p71 ziti-edge-tunnel[15948]: (15948)[   416088.816]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
Feb 21 14:13:17 nixos-p71 ziti-edge-tunnel[15948]: (15948)[   416296.960]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.3] for query[1:$MAC_MACHINE]
Feb 21 14:13:18 nixos-p71 ziti-edge-tunnel[15948]: (15948)[   416297.137]   ERROR ziti-sdk:connect.c:959 connect_reply_cb() conn[0.39/Connecting] failed to connect, reason=service 7Q5ySNwUVQm3UKZtDEMoHn has no terminators
Feb 21 14:13:18 nixos-p71 ziti-edge-tunnel[15948]: (15948)[   416297.137]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed

Ziti version: v0.27.5
Ziti edge tunnel version: v0.20.20

I haven’t noticed yet if this is reproducible. Will update the thread again if so.

2

Well that stinks! But thank you for the post. The main developer of this app is out of the office this week, but I created an Issue to track, and we’ll look for somebody to step in, reproduce, and evaluate the core dump.

1 Like
3

Thank you Dave, appreciate that!

4

Quick update: we think we have a handle on the issue you are seeing. @ekoby is working on a fix today. Thanks again for letting us know.

1 Like