Ziti Edge Tunnel On Linux Leaks Memory

Support General Questions
1

I noticed that ziti-edge-tunnel has a severe memory leak.

The percent memory usage dropped from 70.2% to 31.2% following a restart of ziti-edge-tunnel.

The systemd logs show this just before restarting the service:

Apr 07 07:56:09 computer-name ziti-edge-tunnel[933]: (933)[   207515.754]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3253]
Apr 07 07:56:19 computer-name ziti-edge-tunnel[933]: (933)[   207525.756]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:40962 err=-14, terminating connection
Apr 07 07:56:19 computer-name ziti-edge-tunnel[933]: (933)[   207525.756]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3254]
Apr 07 07:56:29 computer-name ziti-edge-tunnel[933]: (933)[   207535.757]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:51958 err=-14, terminating connection
Apr 07 07:56:29 computer-name ziti-edge-tunnel[933]: (933)[   207535.757]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3255]
Apr 07 07:56:39 computer-name ziti-edge-tunnel[933]: (933)[   207545.758]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:53044 err=-14, terminating connection
Apr 07 07:56:39 computer-name ziti-edge-tunnel[933]: (933)[   207545.758]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3256]
Apr 07 07:56:49 computer-name ziti-edge-tunnel[933]: (933)[   207555.759]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:35700 err=-14, terminating connection
Apr 07 07:56:49 computer-name ziti-edge-tunnel[933]: (933)[   207555.759]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3257]
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.201]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:36368 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.201]    WARN ziti-sdk:connect.c:222 close_conn_internal() conn[0.3179/lNMEp26N/Closed](jellyfin) dumping 24 bytes of undelivered data
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.202]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:57346 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.202]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3259]
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.253]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:40264 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.253]    WARN ziti-sdk:connect.c:222 close_conn_internal() conn[0.3221/UNJ3fWNo/Closed](jellyfin) dumping 13026546579 bytes of undelivered data
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:43354 err=-14, terminating connection
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3118]
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3118]

Here is a relevant screenshot from the output of glances:
Ziti Tunnel High Memory-pre-restart

2

Which version are you running? We did find a leak that was fixed in 1.5.4.1.

3

This was on version 1.4.5.
I just upgraded to version 1.5.4.2, and I will keep an eye on it :slight_smile:

1 Like
4

This was just before restarting the service on version 1.5.4.2
ziti-edge-tunnel-1.5.4.2

The memory useage drops off significantly following a restart of the service.

5

That's very interesting. Are you using one of our builds from GitHub (or the packet repos), or are you building it yourself?

Could you possibly run it under valgrind and share the report? fwiw I have run with valgrind and did not see any leaks, but I may not be exercising your use case.

6

Could you please try ziti-edge-tunnel v1.5.12 if you're still seeing unreasonable footprint growth from ziti-edge-tunnel?

7

I just installed it and I will keep an eye on it :slight_smile:

1 Like