Ziti-Edge-Tunnel crashes with too many open files

1

Hi,
I understand that this is probably not a failure that is (solely) caused by Ziti-Edge-Tunnel, but since it allows for Ziti to crash, I thought it might be good to catch.
One of our Tunnel crashed with a Too many open files error:

Apr 10 03:08:38 zabbix ziti-edge-tunnel[651]: (651)[  7545947.464]   ERROR tunnel-cbs:ziti_hosting.c:658 on_hosted_client_connect_resolved() hosted_service[zabbix_agent.svc], client[company2_s0011.company.ziti] client_src_addr[tcp:100.64.0.1:53557]: uv_tcp_connect failed: too many open files
Apr 10 03:08:38 zabbix ziti-edge-tunnel[651]: (651)[  7545947.464]   ERROR tunnel-cbs:ziti_hosting.c:658 on_hosted_client_connect_resolved() hosted_service[zabbix_agent.svc], client[soar.company.ziti] client_src_addr[tcp:100.64.0.1:46257]: uv_tcp_connect failed: too many open files
Apr 10 03:08:38 zabbix ziti-edge-tunnel[651]: (651)[  7545947.464]   ERROR tunnel-cbs:ziti_hosting.c:658 on_hosted_client_connect_resolved() hosted_service[zabbix_agent.svc], client[zt.company.ziti] client_src_addr[tcp:100.64.0.1:35327]: uv_tcp_connect failed: too many open files
Apr 10 03:08:38 zabbix ziti-edge-tunnel[651]: (651)[  7545947.464]   ERROR tunnel-cbs:ziti_hosting.c:658 on_hosted_client_connect_resolved() hosted_service[zabbix_agent.svc], client[bastion.company.ziti] client_src_addr[tcp:100.64.0.1:51921]: uv_tcp_connect failed: too many open files
Apr 10 03:08:38 zabbix ziti-edge-tunnel[651]: (651)[  7545947.464]   ERROR tunnel-cbs:ziti_hosting.c:658 on_hosted_client_connect_resolved() hosted_service[zabbix_agent.svc], client[el01.company.ziti] client_src_addr[tcp:100.64.0.1:52273]: uv_tcp_connect failed: too many open files

We are unsure if the error was just caused because of too many files open or if Ziti caused the many open files. Either way it might be nice to check the limit and warn if it's getting close?
I do assume it was a soft/hard limit that was on a user level of the user ziti, under which ziti-edge-tunnel is run.

2

Thanks for bringing this up. We haven't seen problems with ziti-edge-tunnel leaking connections recently, but it is possible. Even if it isn't leaking connections, it shouldn't crash when it runs out of file descriptors.

  • Which version of zit-edge-tunnel are you seeing this with?
  • Do you know what the file descriptor limit is for the ziti-edge-tunnel process? Would it make sense that your hosting tunneler is handling roughly that many connections

If you see this again could you get a report of the active openziti connections by running sudo ziti-edge-tunnel dump -p /path/to/directory on the host that's running the tunneler. The directory will be populated with a ".ziti" file for each identity showing the currently handled services and connections.

You can also get an idea of the active underlay connections by looking at ls -l /proc/PID/fd (where PID is the process ID of the ziti-edge-tunnel process).

So, it's possible that ziti-edge-tunnel isn't always releasing connections when they are closed, but it's also possible that you are simply trying to handle more connections than the ulimit allows. It's also possible that your client and/or server are not closing correctly (ziti-edge-tunnel will keep a connection open until it sees a close / FIN from both sides of the connection).

1 Like