We're running quite a couple of Ziti tunnelers which we want to keep running also on network disconnects that are not caused by Ziti. Right now, we sometimes get situations where the tunneler gives up after some time which is not what we want. Instead, I'd like the tunneler to continue endlessly until it can reach controller and/or routers.
I'm sure there are some retry counters we need to adjust here. Which are those? Are there pitfalls?
Here's some of the errors we get, after which the tunnelers aren't connecting anymore:
Occasion 1:
Mai 02 01:48:32 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671243.133] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281324/HqwtJ1uO/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:32 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671243.133] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.559] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281326/8-vhI5J-/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.559] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.559] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281325/7bPQmF9T/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.559] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281330/inSbVX7l/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281329/vOQN2wy0/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281327/ArLoqy-O/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281328/zTmUyXb1/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:47 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671258.560] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:49 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671260.142] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281331/fAcvG1Q5/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:49 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671260.142] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:48:53 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671261.134] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.281332/ei8k2jI1/Connecting] failed to connect, reason=timeout waiting for message reply: context deadline exceeded
Mai 02 01:48:53 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671261.134] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Mai 02 01:49:01 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671269.234] ERROR tunnel-sdk:tunnel_tcp.c:212 tunneler_tcp_write() failed to tcp_write -11 (936, 936)
Mai 02 01:49:01 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671269.234] ERROR tunnel-cbs:ziti_tunnel_cbs.c:121 on_ziti_data() failed to write to client
Mai 02 01:49:01 srv1 ziti-edge-tunnel[3282449]: (3282449)[ 671269.234] WARN ziti-sdk:connect.c:820 flush_to_client() conn[0.281337/o68Hy9J0/Closed] client indicated error[-1] accepting data (0 bytes buffered)
Occasion 2:
May 02 03:12:10 srv2 ziti-edge-tunnel[385593]: (385593)[ 797022.866] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.254261/Z0mhhwzq/Connecting] failed to connect, reason=exceeded maximum [2] retries creating >
May 02 03:12:10 srv2 ziti-edge-tunnel[385593]: (385593)[ 797022.866] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
May 02 03:12:11 srv2 ziti-edge-tunnel[385593]: (385593)[ 797023.937] ERROR tunnel-sdk:tunnel_tcp.c:188 on_tcp_client_err() client=tcp:100.64.0.1:39049 err=-14, terminating connection
May 02 03:12:11 srv2 ziti-edge-tunnel[385593]: (385593)[ 797023.945] ERROR tunnel-sdk:tunnel_tcp.c:188 on_tcp_client_err() client=tcp:100.64.0.1:56069 err=-14, terminating connection
May 02 03:12:11 srv2 ziti-edge-tunnel[385593]: (385593)[ 797023.946] WARN ziti-sdk:channel.c:524 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED72] conn_id[254263]
May 02 03:12:11 srv2 ziti-edge-tunnel[385593]: (385593)[ 797023.948] WARN ziti-sdk:channel.c:524 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED72] conn_id[254262]
May 02 03:12:11 srv2 ziti-edge-tunnel[385593]: (385593)[ 797023.949] ERROR tunnel-sdk:tunnel_tcp.c:188 on_tcp_client_err() client=tcp:100.64.0.1:44757 err=-14, terminating connection