Ziti client not reconnecting

Tunneler Apps Ziti Desktop Edge for Windows
1

I have Ziti installed at a customer site. I have the controller/edge router in an Oracle cloud instance. All was working well, until I got a message about them turning it off due to not pulling enough load on the instance. I then found an article and used SysBench to artificially stress the CPU during the night. That went a bit peer shape as (LOL) instance was UTC, so it ran from 2PM → 6PM. We were getting users complaining about inability to connect to endpoints, or services disappearing out of the client.

Anyway, I changed from that to another application, LookBusy which runs the processors at 25% CPU constantly, which gives me plenty of headroom, and the change has bought stability to Ziti again.

However, what I have been seeing, is that clients (ZDEW and Linux) don’t seem to reconnect when the controller comes back online. This may be offline for a few days. This has caused issues when the device is remote.

I finally saw the issue exhibited itself today. I could see the source and destination computer status.
On my client, I could see the service was available. On the destination computer, it showed me two services (normal). However, I could not connect to the service on this endpoint (RDP). All appeared to be normal when looking at the destination client.

The only way to get the tunnel to work was to restart the destination client (App: 2.1.16.0). So, I have enclosed a number of the logs from the line, and uploaded to support for your thoughts. Others have been using other endpoints without issue, so it appears that it was just this endpoint client that was causing an issue.

You can see when I reset the client in the following logs. Once restarted, I could connect to the client. Also,

[2023-07-11T02:48:29.417Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-11T02:48:29.417Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 1
[2023-07-11T02:48:29.609Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-11T02:48:29.609Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-11T02:48:29.609Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 1
[2023-07-11T02:48:34.374Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:48:54.378Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[2]
[2023-07-11T02:49:03.377Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:49:38.035Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[3]
[2023-07-11T02:50:08.087Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:50:42.288Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[4]
[2023-07-11T02:50:44.130Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:51:02.331Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[5]
[2023-07-11T02:51:04.057Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:51:25.324Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[6]
[2023-07-11T02:51:28.068Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:52:24.887Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[7]
[2023-07-11T02:53:06.603Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T02:53:09.564Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[8]
[2023-07-11T03:13:46.515Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T03:13:46.766Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[9]
[2023-07-11T03:13:49.966Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T03:13:59.186Z]   ERROR ziti-sdk:channel.c:488 dispatch_message() ch[0] received message without conn_id or for unknown connection ct[ED71] conn_id[10]
[2023-07-11T04:44:00.221Z]   ERROR ziti-sdk:connect.c:886 conn_inbound_data_msg() D:/a/desktop-edge-win/desktop-edge-win/ziti-edge-tunnel/deps/ziti-tunneler-sdk-c/build/_deps/ziti-sdk-c-src/library/connect.c:877 - crypto_secretstream_xchacha20poly1305_pull(&conn->crypt_i, plain_text, &plain_len, &tag, msg->body, msg->header.body_len, NULL, 0) => -1 (Unknown error)
[2023-07-11T04:44:15.680Z]   ERROR tunnel-cbs:ziti_hosting.c:342 on_hosted_client_connect() incoming connection to service[hdc-jump01.rdp.svc] failed: ziti context is disabled
[2023-07-11T04:44:15.680Z]   ERROR tunnel-cbs:ziti_hosting.c:342 on_hosted_client_connect() incoming connection to service[rdpSvc] failed: ziti context is disabled
[2023-07-11T04:44:16.043Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: ziti context is disabled
[2023-07-11T04:44:16.043Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to ziti context is disabled
[2023-07-11T04:44:24.593Z]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=3/INFO
[2023-07-11T04:44:26.797Z]    INFO ziti-edge-tunnel:instance-config.c:86 load_tunnel_status_from_file() Loading config file from C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/config.json
[2023-07-11T04:44:26.801Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1880 run() ============================ service begins ================================
[2023-07-11T04:44:26.801Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1881 run() Logger initialization
[2023-07-11T04:44:26.801Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1882 run() 	- initialized at   : Tue Jul 11 2023, 16:44:26 PM (local time), 2023-07-11T04:44:26 (UTC)
[2023-07-11T04:44:26.801Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1883 run() 	- log file location: C:\Program Files (x86)\NetFoundry, Inc\Ziti Desktop Edge\/logs/service/ziti-tunneler.log.202307110000.log
[2023-07-11T04:44:26.802Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1884 run() ============================================================================
[2023-07-11T04:47:12.143Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-11T04:47:12.143Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 1
[2023-07-11T04:47:17.128Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-11T04:47:17.128Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-11T04:47:17.128Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 1
[2023-07-11T05:07:03.119Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-11T05:07:03.119Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 1
[2023-07-11T05:07:03.577Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-11T05:07:03.577Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-11T05:07:03.577Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 1
[2023-07-11T10:30:05.306Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-11T10:30:05.306Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-11T10:30:05.306Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/software caused connection abort] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-11T10:30:05.306Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
[2023-07-11T10:30:05.306Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to Ziti Controller is not available

I have just been looking at my logs on my laptop, and getting a lot of these:

[2023-07-11T10:28:20.951Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[1] failed to get service updates err[CONTROLLER_UNAVAILABLE/unknown node or service] from ctrl[https://ziticontroller.thesmithcave.nz:1280]

although I don’t ever seem to be having issues with home controller (on a workstation on all the time) although log seems to be filled with these CONTROLLER_UNAVAILABLE.

Could there be another reason for these?

2

Hello. I think there may be a few things going on here.

First, a connected IPC client closes. I might assume the closing client is the UI, but apparently there were two active clients when this close happened (since the count: 1 is after the close). Do you know what might be using the IPC socket on this host?

Regardless of what the IPC client(s) might do, it obviously shouldn’t cause problems for subsequent connections to the controller. Looking at the code that handle IPC command I see a bug. For one thing, the code shouldn’t try to send a response to a client that closed. Secondly, and more significantly the buffer that contained the incoming command is free’d unconditionally, and the libuv read callback docs say that the buffer may not be valid when the bytes read is less than zero (which is the case for EOF). Now, I tested EOF conditions on macOS and linux and the buffer is valid when a client closes, but the doc is a little squishy and you’re on Windows so things might be different there.

If the buffer is in fact being free’d when it shouldn’t then I’d expect a flat out crash, but I’m not 100% sure how the windows heap deals with this. Are these controller connection errors (those that you see on the customer’s host) always preceded by an IPC client closing?

Now to the connection failures you’re seeing on your home network - the “unknown node or service” indicates a dns lookup failure. Does “ziticontroller.thesmithcave.nz” resolve successfully for you when these connection failures are happening?

3

Thanks for the reply.

I haven't noticed any connection failures when going to use it. Just noticed them in the log and wonder what could be causing these.

I would have to say no. So, from the 9th I have these (not all, but enough to show what is happening)

[2023-07-09T16:32:59.527Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T16:32:59.527Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T16:32:59.527Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/software caused connection abort] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-09T16:32:59.527Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
[2023-07-09T16:32:59.527Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to Ziti Controller is not available
[2023-07-09T21:43:13.798Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T21:43:13.798Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T21:43:13.798Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/software caused connection abort] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-09T21:43:13.798Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
[2023-07-09T21:43:13.798Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to Ziti Controller is not available
[2023-07-09T21:43:33.890Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T21:43:33.890Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4079(software caused connection abort)
[2023-07-09T21:43:33.890Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/software caused connection abort] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-09T21:43:35.351Z]   ERROR ziti-sdk:channel.c:582 latency_timeout() ch[0] no read/write traffic on channel since before latency probe was sent, closing channel

On the 8th, we had something similar, though I dont think I kicked the service

[2023-07-08T16:30:00.085Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4095(end of file)
[2023-07-08T16:30:00.085Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/end of file] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-08T16:30:00.085Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
[2023-07-08T16:30:00.085Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to Ziti Controller is not available
[2023-07-08T16:40:38.466Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-08T16:40:38.466Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 4
[2023-07-08T16:40:38.466Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-08T16:40:38.466Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 3
[2023-07-08T16:40:38.481Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-08T16:40:38.481Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 2
[2023-07-08T16:40:38.506Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-08T16:40:38.506Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 1
[2023-07-08T16:40:40.931Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-08T16:40:40.931Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-08T16:40:40.931Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-08T16:40:40.931Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-08T16:40:40.931Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-08T16:40:40.931Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-08T16:40:40.931Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-08T16:40:40.931Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-08T16:40:40.931Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 1
[2023-07-08T16:56:55.807Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-08T16:56:55.808Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 0
[2023-07-08T16:56:56.271Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-08T16:56:56.271Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-08T16:56:56.271Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 0
[2023-07-08T16:59:36.451Z]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=3/INFO
[2023-07-08T17:00:08.541Z]    INFO ziti-edge-tunnel:instance-config.c:86 load_tunnel_status_from_file() Loading config file from C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/config.json
[2023-07-08T17:00:08.636Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1880 run() ============================ service begins ================================
[2023-07-08T17:00:08.636Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1881 run() Logger initialization
[2023-07-08T17:00:08.636Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1882 run() 	- initialized at   : Sun Jul 09 2023, 05:00:08 AM (local time), 2023-07-08T17:00:08 (UTC)
[2023-07-08T17:00:08.636Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1883 run() 	- log file location: C:\Program Files (x86)\NetFoundry, Inc\Ziti Desktop Edge\/logs/service/ziti-tunneler.log.202307080000.log
[2023-07-08T17:00:08.636Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1884 run() ============================================================================

[2023-07-07T02:55:11.131Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:620 on_cmd() received from client - EOF. Closing connection.
[2023-07-07T02:55:11.131Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:632 on_cmd() IPC client connection closed, count: 4
[2023-07-07T02:55:13.415Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:774 send_events_message() Events client write operation failed, received error - EPIPE
[2023-07-07T02:55:13.415Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:779 send_events_message() Events client connection closed
[2023-07-07T02:55:13.415Z]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:785 send_events_message() Events client connection current count : 4
[2023-07-07T20:29:57.627Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4095(end of file)
[2023-07-07T20:29:57.627Z]   ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ziticontroller.horowhenua.govt.nz] request failed: -4095(end of file)
[2023-07-07T20:29:57.627Z]   ERROR ziti-sdk:ziti.c:1051 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/end of file] from ctrl[https://ziticontroller.horowhenua.govt.nz:443]
[2023-07-07T20:29:57.627Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
[2023-07-07T20:29:57.627Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1193 on_event() ztx[C:\Windows\system32\config\systemprofile\AppData\Roaming/NetFoundry/id.json] failed to connect to controller due to Ziti Controller is not available

Now, on the controller, which is not what I provided before, I am seeing a lot of TLS handshake errors from the IP address where at least 3 or more ZDEW clients exist, so hard to understand what agent is causing it, other than matching timestamps. These are all the errors from that IP address from the 7th

...
{"log":"[5784428.383]    INFO : http: TLS handshake error from 131.203.187.108:62943: EOF\n","stream":"stderr","time":"2023-07-07T02:24:03.343832094Z"}
{"log":"[5786237.241]    INFO : http: TLS handshake error from 131.203.187.108:64516: EOF\n","stream":"stderr","time":"2023-07-07T02:54:12.201954975Z"}
{"log":"[5788041.732]    INFO : http: TLS handshake error from 131.203.187.108:16313: EOF\n","stream":"stderr","time":"2023-07-07T03:24:16.693583659Z"}
{"log":"[5789845.561]    INFO : http: TLS handshake error from 131.203.187.108:62163: EOF\n","stream":"stderr","time":"2023-07-07T03:54:20.521909828Z"}
{"log":"[5791646.303]    INFO : http: TLS handshake error from 131.203.187.108:6976: EOF\n","stream":"stderr","time":"2023-07-07T04:24:21.263701497Z"}
{"log":"[5793456.798]    INFO : http: TLS handshake error from 131.203.187.108:64585: EOF\n","stream":"stderr","time":"2023-07-07T04:54:31.759802128Z"}
{"log":"[5795261.346]    INFO : http: TLS handshake error from 131.203.187.108:20548: EOF\n","stream":"stderr","time":"2023-07-07T05:24:36.307027068Z"}
{"log":"[5797064.471]    INFO : http: TLS handshake error from 131.203.187.108:24404: EOF\n","stream":"stderr","time":"2023-07-07T05:54:39.432055959Z"}
{"log":"[5798872.079]    INFO : http: TLS handshake error from 131.203.187.108:16440: EOF\n","stream":"stderr","time":"2023-07-07T06:24:47.040092534Z"}
{"log":"[5800674.171]    INFO : http: TLS handshake error from 131.203.187.108:23921: EOF\n","stream":"stderr","time":"2023-07-07T06:54:49.131971591Z"}
{"log":"[5802476.469]    INFO : http: TLS handshake error from 131.203.187.108:6227: EOF\n","stream":"stderr","time":"2023-07-07T07:24:51.430241205Z"}
{"log":"[5804278.639]    INFO : http: TLS handshake error from 131.203.187.108:14598: EOF\n","stream":"stderr","time":"2023-07-07T07:54:53.600233256Z"}
{"log":"[5806080.856]    INFO : http: TLS handshake error from 131.203.187.108:18859: EOF\n","stream":"stderr","time":"2023-07-07T08:24:55.817527823Z"}
{"log":"[5807883.037]    INFO : http: TLS handshake error from 131.203.187.108:22468: EOF\n","stream":"stderr","time":"2023-07-07T08:54:57.997946262Z"}
{"log":"[5809684.971]    INFO : http: TLS handshake error from 131.203.187.108:19726: EOF\n","stream":"stderr","time":"2023-07-07T09:24:59.932241955Z"}
{"log":"[5811487.158]    INFO : http: TLS handshake error from 131.203.187.108:63235: EOF\n","stream":"stderr","time":"2023-07-07T09:55:02.11942233Z"}
{"log":"[5813289.213]    INFO : http: TLS handshake error from 131.203.187.108:16521: EOF\n","stream":"stderr","time":"2023-07-07T10:25:04.173710496Z"}
{"log":"[5815091.639]    INFO : http: TLS handshake error from 131.203.187.108:12096: EOF\n","stream":"stderr","time":"2023-07-07T10:55:06.599762845Z"}
{"log":"[5816893.730]    INFO : http: TLS handshake error from 131.203.187.108:65140: EOF\n","stream":"stderr","time":"2023-07-07T11:25:08.691039338Z"}
{"log":"[5818695.623]    INFO : http: TLS handshake error from 131.203.187.108:65471: EOF\n","stream":"stderr","time":"2023-07-07T11:55:10.583566222Z"}
{"log":"[5820497.865]    INFO : http: TLS handshake error from 131.203.187.108:11617: EOF\n","stream":"stderr","time":"2023-07-07T12:25:12.82641013Z"}
{"log":"[5822299.830]    INFO : http: TLS handshake error from 131.203.187.108:14062: EOF\n","stream":"stderr","time":"2023-07-07T12:55:14.791284537Z"}
{"log":"[5824102.265]    INFO : http: TLS handshake error from 131.203.187.108:6914: EOF\n","stream":"stderr","time":"2023-07-07T13:25:17.225966012Z"}
{"log":"[5825904.379]    INFO : http: TLS handshake error from 131.203.187.108:19955: EOF\n","stream":"stderr","time":"2023-07-07T13:55:19.34161651Z"}
{"log":"[5827706.714]    INFO : http: TLS handshake error from 131.203.187.108:12007: EOF\n","stream":"stderr","time":"2023-07-07T14:25:21.674900633Z"}
{"log":"[5829509.764]    INFO : http: TLS handshake error from 131.203.187.108:23150: EOF\n","stream":"stderr","time":"2023-07-07T14:55:24.725055205Z"}
{"log":"[5831311.263]    INFO : http: TLS handshake error from 131.203.187.108:13344: EOF\n","stream":"stderr","time":"2023-07-07T15:25:26.224350702Z"}
{"log":"[5833113.058]    INFO : http: TLS handshake error from 131.203.187.108:12527: EOF\n","stream":"stderr","time":"2023-07-07T15:55:28.018828919Z"}
{"log":"[5834915.317]    INFO : http: TLS handshake error from 131.203.187.108:18932: EOF\n","stream":"stderr","time":"2023-07-07T16:25:30.278270293Z"}
{"log":"[5836718.438]    INFO : http: TLS handshake error from 131.203.187.108:9207: EOF\n","stream":"stderr","time":"2023-07-07T16:55:33.399508489Z"}
{"log":"[5838522.618]    INFO : http: TLS handshake error from 131.203.187.108:6962: EOF\n","stream":"stderr","time":"2023-07-07T17:25:37.578988829Z"}
{"log":"[5840332.271]    INFO : http: TLS handshake error from 131.203.187.108:19641: EOF\n","stream":"stderr","time":"2023-07-07T17:55:47.232609889Z"}
{"log":"[5842134.736]    INFO : http: TLS handshake error from 131.203.187.108:7014: EOF\n","stream":"stderr","time":"2023-07-07T18:25:49.697277675Z"}
{"log":"[5843936.521]    INFO : http: TLS handshake error from 131.203.187.108:8267: EOF\n","stream":"stderr","time":"2023-07-07T18:55:51.481710825Z"}
{"log":"[5845738.274]    INFO : http: TLS handshake error from 131.203.187.108:15798: EOF\n","stream":"stderr","time":"2023-07-07T19:25:53.237115052Z"}
{"log":"[5847544.676]    INFO : http: TLS handshake error from 131.203.187.108:22886: EOF\n","stream":"stderr","time":"2023-07-07T19:55:59.637671021Z"}
{"log":"[5849349.993]    INFO : http: TLS handshake error from 131.203.187.108:10213: EOF\n","stream":"stderr","time":"2023-07-07T20:26:04.954510479Z"}
{"log":"[5849581.688]    INFO : http: TLS handshake error from 131.203.187.108:61224: read tcp 172.20.0.2:443-\u003e131.203.187.108:61224: i/o timeout\n","stream":"stderr","time":"2023-07-07T20:29:56.650285796Z"}
{"log":"[5851152.119]    INFO : http: TLS handshake error from 131.203.187.108:64977: EOF\n","stream":"stderr","time":"2023-07-07T20:56:07.080585861Z"}
{"log":"[5852954.382]    INFO : http: TLS handshake error from 131.203.187.108:6098: EOF\n","stream":"stderr","time":"2023-07-07T21:26:09.343703872Z"}
{"log":"[5854756.541]    INFO : http: TLS handshake error from 131.203.187.108:64536: EOF\n","stream":"stderr","time":"2023-07-07T21:56:11.502367801Z"}
{"log":"[5856558.693]    INFO : http: TLS handshake error from 131.203.187.108:8652: EOF\n","stream":"stderr","time":"2023-07-07T22:26:13.653913379Z"}
{"log":"[5858361.047]    INFO : http: TLS handshake error from 131.203.187.108:18684: EOF\n","stream":"stderr","time":"2023-07-07T22:56:16.008133083Z"}
{"log":"[5860162.868]    INFO : http: TLS handshake error from 131.203.187.108:5117: EOF\n","stream":"stderr","time":"2023-07-07T23:26:17.828939278Z"}
{"log":"[5861965.308]    INFO : http: TLS handshake error from 131.203.187.108:63964: EOF\n","stream":"stderr","time":"2023-07-07T23:56:20.269347405Z"}
{"log":"[5863767.473]    INFO : http: TLS handshake error from 131.203.187.108:12232: EOF\n","stream":"stderr","time":"2023-07-08T00:26:22.434607873Z"}
{"log":"[5865569.339]    INFO : http: TLS handshake error from 131.203.187.108:8889: EOF\n","stream":"stderr","time":"2023-07-08T00:56:24.300501269Z"}
{"log":"[5867371.995]    INFO : http: TLS handshake error from 131.203.187.108:24605: EOF\n","stream":"stderr","time":"2023-07-08T01:26:26.955854347Z"}
{"log":"[5869181.350]    INFO : http: TLS handshake error from 131.203.187.108:14124: EOF\n","stream":"stderr","time":"2023-07-08T01:56:36.311174673Z"}
{"log":"[5870984.061]    INFO : http: TLS handshake error from 131.203.187.108:62420: EOF\n","stream":"stderr","time":"2023-07-08T02:26:39.022546521Z"}
{"log":"[5872786.638]    INFO : http: TLS handshake error from 131.203.187.108:12751: EOF\n","stream":"stderr","time":"2023-07-08T02:56:41.599238194Z"}
{"log":"[5874590.402]    INFO : http: TLS handshake error from 131.203.187.108:13201: EOF\n","stream":"stderr","time":"2023-07-08T03:26:45.363601709Z"}
{"log":"[5876392.112]    INFO : http: TLS handshake error from 131.203.187.108:10641: EOF\n","stream":"stderr","time":"2023-07-08T03:56:47.072802032Z"}
{"log":"[5878195.075]    INFO : http: TLS handshake error from 131.203.187.108:24171: EOF\n","stream":"stderr","time":"2023-07-08T04:26:50.035934232Z"}
{"log":"[5880000.303]    INFO : http: TLS handshake error from 131.203.187.108:65181: EOF\n","stream":"stderr","time":"2023-07-08T04:56:55.266227462Z"}
{"log":"[5881810.999]    INFO : http: TLS handshake error from 131.203.187.108:8528: EOF\n","stream":"stderr","time":"2023-07-08T05:27:05.960112206Z"}
{"log":"[5883613.787]    INFO : http: TLS handshake error from 131.203.187.108:18630: EOF\n","stream":"stderr","time":"2023-07-08T05:57:08.748382248Z"}
{"log":"[5885417.483]    INFO : http: TLS handshake error from 131.203.187.108:17957: EOF\n","stream":"stderr","time":"2023-07-08T06:27:12.443804737Z"}
{"log":"[5887219.493]    INFO : http: TLS handshake error from 131.203.187.108:63510: EOF\n","stream":"stderr","time":"2023-07-08T06:57:14.454628883Z"}
{"log":"[5889023.145]    INFO : http: TLS handshake error from 131.203.187.108:20708: EOF\n","stream":"stderr","time":"2023-07-08T07:27:18.106157291Z"}
{"log":"[5890826.595]    INFO : http: TLS handshake error from 131.203.187.108:13985: EOF\n","stream":"stderr","time":"2023-07-08T07:57:21.556110128Z"}
{"log":"[5892631.248]    INFO : http: TLS handshake error from 131.203.187.108:24074: EOF\n","stream":"stderr","time":"2023-07-08T08:27:26.209247894Z"}
{"log":"[5894441.887]    INFO : http: TLS handshake error from 131.203.187.108:64404: EOF\n","stream":"stderr","time":"2023-07-08T08:57:36.847901016Z"}
{"log":"[5896245.839]    INFO : http: TLS handshake error from 131.203.187.108:8145: EOF\n","stream":"stderr","time":"2023-07-08T09:27:40.800173727Z"}
{"log":"[5896385.479]    INFO : http: TLS handshake error from 131.203.187.108:50293: read tcp 172.20.0.2:443-\u003e131.203.187.108:50293: i/o timeout\n","stream":"stderr","time":"2023-07-08T09:30:00.441108637Z"}
{"log":"[5898047.803]    INFO : http: TLS handshake error from 131.203.187.108:17003: EOF\n","stream":"stderr","time":"2023-07-08T09:57:42.763896268Z"}
{"log":"[5899849.875]    INFO : http: TLS handshake error from 131.203.187.108:17591: EOF\n","stream":"stderr","time":"2023-07-08T10:27:44.83620445Z"}
{"log":"[5901652.078]    INFO : http: TLS handshake error from 131.203.187.108:21659: EOF\n","stream":"stderr","time":"2023-07-08T10:57:47.039160896Z"}
{"log":"[5903454.253]    INFO : http: TLS handshake error from 131.203.187.108:14432: EOF\n","stream":"stderr","time":"2023-07-08T11:27:49.214267727Z"}
{"log":"[5905256.635]    INFO : http: TLS handshake error from 131.203.187.108:11521: EOF\n","stream":"stderr","time":"2023-07-08T11:57:51.595834047Z"}
{"log":"[5907058.464]    INFO : http: TLS handshake error from 131.203.187.108:61916: EOF\n","stream":"stderr","time":"2023-07-08T12:27:53.425016616Z"}
{"log":"[5908860.552]    INFO : http: TLS handshake error from 131.203.187.108:12376: EOF\n","stream":"stderr","time":"2023-07-08T12:57:55.513450474Z"}
{"log":"[5910662.874]    INFO : http: TLS handshake error from 131.203.187.108:65530: EOF\n","stream":"stderr","time":"2023-07-08T13:27:57.835379144Z"}
{"log":"[5912465.106]    INFO : http: TLS handshake error from 131.203.187.108:62418: EOF\n","stream":"stderr","time":"2023-07-08T13:58:00.066965505Z"}
{"log":"[5914267.341]    INFO : http: TLS handshake error from 131.203.187.108:18702: EOF\n","stream":"stderr","time":"2023-07-08T14:28:02.301791633Z"}
{"log":"[5916070.474]    INFO : http: TLS handshake error from 131.203.187.108:19359: EOF\n","stream":"stderr","time":"2023-07-08T14:58:05.435021922Z"}
{"log":"[5917871.634]    INFO : http: TLS handshake error from 131.203.187.108:61727: EOF\n","stream":"stderr","time":"2023-07-08T15:28:06.59549562Z"}
{"log":"[5919673.640]    INFO : http: TLS handshake error from 131.203.187.108:10585: EOF\n","stream":"stderr","time":"2023-07-08T15:58:08.601216203Z"}
{"log":"[5921475.794]    INFO : http: TLS handshake error from 131.203.187.108:22288: EOF\n","stream":"stderr","time":"2023-07-08T16:28:10.754709566Z"}
{"log":"[5921583.751]    INFO : http: TLS handshake error from 131.203.187.108:53219: read tcp 172.20.0.2:443-\u003e131.203.187.108:53219: i/o timeout\n","stream":"stderr","time":"2023-07-08T16:29:58.711873042Z"}
{"log":"[5923438.844]    INFO : http: TLS handshake error from 131.203.187.108:12930: EOF\n","stream":"stderr","time":"2023-07-08T17:00:53.804727756Z"}
{"log":"[5925243.847]    INFO : http: TLS handshake error from 131.203.187.108:65093: EOF\n","stream":"stderr","time":"2023-07-08T17:30:58.807717455Z"}
{"log":"[5927045.937]    INFO : http: TLS handshake error from 131.203.187.108:21033: EOF\n","stream":"stderr","time":"2023-07-08T18:01:00.89769482Z"}
{"log":"[5928848.353]    INFO : http: TLS handshake error from 131.203.187.108:12250: EOF\n","stream":"stderr","time":"2023-07-08T18:31:03.314090793Z"}
{"log":"[5930650.884]    INFO : http: TLS handshake error from 131.203.187.108:23659: EOF\n","stream":"stderr","time":"2023-07-08T19:01:05.846998109Z"}
{"log":"[5932457.474]    INFO : http: TLS handshake error from 131.203.187.108:9576: EOF\n","stream":"stderr","time":"2023-07-08T19:31:12.435679991Z"}
{"log":"[5932772.005]    INFO : http: TLS handshake error from 131.203.187.108:50768: read tcp 172.20.0.2:443-\u003e131.203.187.108:50768: i/o timeout\n","stream":"stderr","time":"2023-07-08T19:36:26.966095656Z"}
{"log":"[5934265.172]    INFO : http: TLS handshake error from 131.203.187.108:61552: EOF\n","stream":"stderr","time":"2023-07-08T20:01:20.133495231Z"}
{"log":"[5936071.508]    INFO : http: TLS handshake error from 131.203.187.108:10992: EOF\n","stream":"stderr","time":"2023-07-08T20:31:26.469334085Z"}
{"log":"[5937879.683]    INFO : http: TLS handshake error from 131.203.187.108:9878: EOF\n","stream":"stderr","time":"2023-07-08T21:01:34.644058084Z"}
{"log":"[5939681.871]    INFO : http: TLS handshake error from 131.203.187.108:22053: EOF\n","stream":"stderr","time":"2023-07-08T21:31:36.832923214Z"}
{"log":"[5941484.080]    INFO : http: TLS handshake error from 131.203.187.108:17404: EOF\n","stream":"stderr","time":"2023-07-08T22:01:39.041034139Z"}
{"log":"[5943290.237]    INFO : http: TLS handshake error from 131.203.187.108:16465: EOF\n","stream":"stderr","time":"2023-07-08T22:31:45.198121872Z"}
{"log":"[5945099.265]    INFO : http: TLS handshake error from 131.203.187.108:17035: EOF\n","stream":"stderr","time":"2023-07-08T23:01:54.226294027Z"}
{"log":"[5946901.348]    INFO : http: TLS handshake error from 131.203.187.108:65330: EOF\n","stream":"stderr","time":"2023-07-08T23:31:56.308737372Z"}
{"log":"[5948703.936]    INFO : http: TLS handshake error from 131.203.187.108:19664: EOF\n","stream":"stderr","time":"2023-07-09T00:01:58.897530638Z"}
{"log":"[5950510.871]    INFO : http: TLS handshake error from 131.203.187.108:17450: EOF\n","stream":"stderr","time":"2023-07-09T00:32:05.835842061Z"}
{"log":"[5952317.949]    INFO : http: TLS handshake error from 131.203.187.108:11563: EOF\n","stream":"stderr","time":"2023-07-09T01:02:12.91062353Z"}
{"log":"[5954121.171]    INFO : http: TLS handshake error from 131.203.187.108:12851: EOF\n","stream":"stderr","time":"2023-07-09T01:32:16.13195867Z"}
{"log":"[5955932.470]    INFO : http: TLS handshake error from 131.203.187.108:23441: EOF\n","stream":"stderr","time":"2023-07-09T02:02:27.431266522Z"}
{"log":"[5957735.015]    INFO : http: TLS handshake error from 131.203.187.108:16031: EOF\n","stream":"stderr","time":"2023-07-09T02:32:29.9764849Z"}
{"log":"[5959538.271]    INFO : http: TLS handshake error from 131.203.187.108:22117: EOF\n","stream":"stderr","time":"2023-07-09T03:02:33.232167086Z"}
{"log":"[5961344.508]    INFO : http: TLS handshake error from 131.203.187.108:6812: EOF\n","stream":"stderr","time":"2023-07-09T03:32:39.469550075Z"}
{"log":"[5963151.371]    INFO : http: TLS handshake error from 131.203.187.108:13340: EOF\n","stream":"stderr","time":"2023-07-09T04:02:46.331992166Z"}
{"log":"[5964953.351]    INFO : http: TLS handshake error from 131.203.187.108:8474: EOF\n","stream":"stderr","time":"2023-07-09T04:32:48.31200842Z"}
{"log":"[5966757.202]    INFO : http: TLS handshake error from 131.203.187.108:13581: EOF\n","stream":"stderr","time":"2023-07-09T05:02:52.163119188Z"}
{"log":"[5968559.525]    INFO : http: TLS handshake error from 131.203.187.108:10650: EOF\n","stream":"stderr","time":"2023-07-09T05:32:54.485656853Z"}
{"log":"[5970370.477]    INFO : http: TLS handshake error from 131.203.187.108:10246: EOF\n","stream":"stderr","time":"2023-07-09T06:03:05.438100134Z"}
{"log":"[5972180.232]    INFO : http: TLS handshake error from 131.203.187.108:23010: EOF\n","stream":"stderr","time":"2023-07-09T06:33:15.193444758Z"}
{"log":"[5973982.137]    INFO : http: TLS handshake error from 131.203.187.108:6132: EOF\n","stream":"stderr","time":"2023-07-09T07:03:17.097806986Z"}
{"log":"[5975784.269]    INFO : http: TLS handshake error from 131.203.187.108:10572: EOF\n","stream":"stderr","time":"2023-07-09T07:33:19.230961545Z"}
{"log":"[5977586.709]    INFO : http: TLS handshake error from 131.203.187.108:23735: EOF\n","stream":"stderr","time":"2023-07-09T08:03:21.670531997Z"}
{"log":"[5979389.712]    INFO : http: TLS handshake error from 131.203.187.108:15680: EOF\n","stream":"stderr","time":"2023-07-09T08:33:24.673653643Z"}
{"log":"[5981190.732]    INFO : http: TLS handshake error from 131.203.187.108:10110: EOF\n","stream":"stderr","time":"2023-07-09T09:03:25.693300671Z"}
{"log":"[5982993.034]    INFO : http: TLS handshake error from 131.203.187.108:22982: EOF\n","stream":"stderr","time":"2023-07-09T09:33:27.99548873Z"}
{"log":"[5984795.572]    INFO : http: TLS handshake error from 131.203.187.108:16300: EOF\n","stream":"stderr","time":"2023-07-09T10:03:30.535349321Z"}
{"log":"[5986599.679]    INFO : http: TLS handshake error from 131.203.187.108:8727: EOF\n","stream":"stderr","time":"2023-07-09T10:33:34.640533403Z"}
{"log":"[5988409.639]    INFO : http: TLS handshake error from 131.203.187.108:15095: EOF\n","stream":"stderr","time":"2023-07-09T11:03:44.600435947Z"}
{"log":"[5990214.604]    INFO : http: TLS handshake error from 131.203.187.108:13555: EOF\n","stream":"stderr","time":"2023-07-09T11:33:49.565249639Z"}
{"log":"[5992024.937]    INFO : http: TLS handshake error from 131.203.187.108:65117: EOF\n","stream":"stderr","time":"2023-07-09T12:03:59.897997838Z"}
{"log":"[5993826.454]    INFO : http: TLS handshake error from 131.203.187.108:11352: EOF\n","stream":"stderr","time":"2023-07-09T12:34:01.415434069Z"}
{"log":"[5995628.555]    INFO : http: TLS handshake error from 131.203.187.108:10474: EOF\n","stream":"stderr","time":"2023-07-09T13:04:03.516162334Z"}
{"log":"[5997431.374]    INFO : http: TLS handshake error from 131.203.187.108:24265: EOF\n","stream":"stderr","time":"2023-07-09T13:34:06.336670221Z"}
{"log":"[5999233.419]    INFO : http: TLS handshake error from 131.203.187.108:14304: EOF\n","stream":"stderr","time":"2023-07-09T14:04:08.380242594Z"}
{"log":"[6001035.274]    INFO : http: TLS handshake error from 131.203.187.108:65479: EOF\n","stream":"stderr","time":"2023-07-09T14:34:10.234963813Z"}
{"log":"[6002837.239]    INFO : http: TLS handshake error from 131.203.187.108:5447: EOF\n","stream":"stderr","time":"2023-07-09T15:04:12.199693567Z"}
{"log":"[6004642.966]    INFO : http: TLS handshake error from 131.203.187.108:15671: EOF\n","stream":"stderr","time":"2023-07-09T15:34:17.927228728Z"}
{"log":"[6006452.749]    INFO : http: TLS handshake error from 131.203.187.108:7369: EOF\n","stream":"stderr","time":"2023-07-09T16:04:27.710710113Z"}
{"log":"[6008257.571]    INFO : http: TLS handshake error from 131.203.187.108:18119: EOF\n","stream":"stderr","time":"2023-07-09T16:34:32.531983122Z"}
{"log":"[6010066.506]    INFO : http: TLS handshake error from 131.203.187.108:14343: EOF\n","stream":"stderr","time":"2023-07-09T17:04:41.467479136Z"}
{"log":"[6011874.916]    INFO : http: TLS handshake error from 131.203.187.108:5486: EOF\n","stream":"stderr","time":"2023-07-09T17:34:49.877559547Z"}
{"log":"[6013681.297]    INFO : http: TLS handshake error from 131.203.187.108:8638: EOF\n","stream":"stderr","time":"2023-07-09T18:04:56.257837543Z"}
{"log":"[6015487.183]    INFO : http: TLS handshake error from 131.203.187.108:62189: EOF\n","stream":"stderr","time":"2023-07-09T18:35:02.143900937Z"}
{"log":"[6017296.117]    INFO : http: TLS handshake error from 131.203.187.108:13367: EOF\n","stream":"stderr","time":"2023-07-09T19:05:11.077922276Z"}
{"log":"[6019098.512]    INFO : http: TLS handshake error from 131.203.187.108:21265: EOF\n","stream":"stderr","time":"2023-07-09T19:35:13.472976682Z"}
{"log":"[6020907.648]    INFO : http: TLS handshake error from 131.203.187.108:64409: EOF\n","stream":"stderr","time":"2023-07-09T20:05:22.608633456Z"}
{"log":"[6022709.838]    INFO : http: TLS handshake error from 131.203.187.108:16254: EOF\n","stream":"stderr","time":"2023-07-09T20:35:24.799398956Z"}
{"log":"[6024511.935]    INFO : http: TLS handshake error from 131.203.187.108:7081: EOF\n","stream":"stderr","time":"2023-07-09T21:05:26.896551399Z"}
{"log":"[6026314.193]    INFO : http: TLS handshake error from 131.203.187.108:17754: EOF\n","stream":"stderr","time":"2023-07-09T21:35:29.153960492Z"}
{"log":"[6028118.612]    INFO : http: TLS handshake error from 131.203.187.108:61598: EOF\n","stream":"stderr","time":"2023-07-09T22:05:33.572685419Z"}
{"log":"[6029921.249]    INFO : http: TLS handshake error from 131.203.187.108:62408: EOF\n","stream":"stderr","time":"2023-07-09T22:35:36.210500229Z"}

I have just checked the Firewall (Fortigate) and made a change, to see if it was getting a bit too involved in the communications.

4

Did your firewall changes make any difference?

So far your logs have drawn my attention to some problems in the IPC command processing code that should definitely be addressed, but nothing that easily explains why clients would fail to reconnect when the controller comes back online. I’ll keep looking for memory related errors in the IPC command processing.

Which version of ZDEW or ziti-edge-tunnel is this?

Out of curiosity, do you know what IPC clients are connecting to the ZDEW process that you’ve shown logs from here? I’ve seen as many as 4 connected clients, which I should be fine but I think is unusual.

If possible, could you post some logs that show this condition on one of the linux clients, preferably from the time the process starts?

Thanks

5

Hey

So, we are having a three day weekend, so only one working day since the last post. However, the client which had the problem, and the controller (cloud) are now not complaining, ie nothing in the logs around TLS issues/connection issues since I made modifications on the Firewall.

There seems to be a bit of talk around connection issues (or reconnection issues when controller been offline). I have a linux endpoint on the end of a LTE connection that suffered this problem (no firewall in the way): linux client → LTE cellular → OC Controller. This definitely exhibited the connection problem after controller went offline. However, I am waiting for the machine to either come back online, or come back from site so I can get the logs to understand what is/was happening.

6

ZDEW 2.1.15.0
Service: 2.1.16

You mean who is using that service?
I have two service endpoints on this machine which are RDP endpoints - one terminates on this machine, and the other is a gateway router to go to another machine.

When I get a machine back from a remote location, hopefully it will keep the logs enough for me to post this back to you. It may be a week or so before it comes back.

7

Just wondering where you landed on this. Did you manage to find an issue with the IPC command? Wondering if there is likely to be a Windows client update at some point which may address the issue, if you did indead find a problem.

I have the same client go down, but nothing really stuck out in the logs (I can send it through to you again). Restart of the service resolved it, so it is not a Fortigate firewall issue, as this has a let fly rule for communicating with the controller and edge router.

8

Hello,

Thanks for bringing this back into the foreground for me. I did look, and I didn't find any real issues with the IPC command logic. Since you've posted this there has been a fair amount of attention to re-connect issues with the ziti SDK that the ZDEW uses.

The main issue you describe here sounds a lot like the one that was fixed with preserve crypto material between rebinds by ekoby · Pull Request #546 · openziti/ziti-sdk-c · GitHub. This fix, along with other "re-bind" fixes, has been pulled into a ZDEW build that is currently labelled pre-release, but I'm reluctant to share that build with you as we know about some additional issues that have been identified and are being worked on for the next iteration. I'll update this thread when the next prerelease is ready; hopefully early next week, assuming the issues with the SDK have been resolved by then.

11

A post was split to a new topic: Upgrading ziti desktop edge for Windows