Ziti Edge Tunnel Docker container fails/exits

pantmal · September 19, 2024, 11:44am

Hello everyone. My team is using OpenZiti in order to provide connectivity across different machines. We're managing a Ziti Controller that is in version: v0.31.4 with ZAC at: 3.0.3.

We provide the following docker-compose.yaml file for the setting up of a Ziti Edge Tunnel, that will connect to our Ziti Controller:

version: "3.9"
services:
    ziti-tun:
        image: openziti/ziti-edge-tunnel
        devices:
            - /dev/net/tun:/dev/net/tun
        volumes:
            - .:/ziti-edge-tunnel
            - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
        environment:
            - ZITI_IDENTITY_BASENAME=ziti_id
            - PFXLOG_NO_JSON=true              # suppress JSON logging
        network_mode: host
        privileged: true

At first, the Ziti Edge Tunnel connects to our Ziti Controller without issue, and the connection across different machines is properly established. However, we notice that after some time (usually after a few hours or days) the ziti Edge Tunnel container appears to exit, seemingly without reason (thus requiring a manual restart etc).

Is it expected for the ziti-edge-tunnel container to exit abruptly? Have you ever experienced something similar?

The logs don't appear to be the same ones in the exited containers from all of the machines, so they don't offer us much. In any case, I will provide the logs from one of the exited containers:

(9)[   498561.633]    WARN ziti-sdk:bind.c:210 session_cb() server[0.62] failed to get session for service[HOST-tcp-10.10.10.19-6443-INTERCEPTS-tcp-10.10.10.19-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498561.929]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498561.929]    WARN ziti-sdk:bind.c:210 session_cb() server[0.71] failed to get session for service[HOST-tcp-10.10.10.52-6443-INTERCEPTS-tcp-10.10.10.52-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498563.428]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498563.428]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
(9)[   498565.290]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498565.290]    WARN ziti-sdk:bind.c:210 session_cb() server[0.50] failed to get session for service[tcp-10.10.10.55]: -14/UNAUTHORIZED
(9)[   498565.650]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(9)[   498565.650]    WARN ziti-sdk:ziti.c:1598 api_session_cb() ztx[0] failed to get api session from ctrl[https://ztc.euprojects.net:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection timed out
(9)[   498567.048]   ERROR ziti-sdk:channel.c:695 reconnect_cb() ch[0] ziti context is not fully authenticated (api_session_state[0]), delaying re-connect
(9)[   498567.048]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 123356ms (attempt = 205)
(9)[   498567.645]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498567.645]    WARN ziti-sdk:bind.c:210 session_cb() server[0.71] failed to get session for service[HOST-tcp-10.10.10.52-6443-INTERCEPTS-tcp-10.10.10.52-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498570.650]    INFO ziti-sdk:ziti.c:914 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://ztc.euprojects.net:8441] api_session_status[0] api_session_expired[TRUE]
(9)[   498570.682]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498570.682]    WARN ziti-sdk:bind.c:210 session_cb() server[0.5] failed to get session for service[internal-svc-k8s]: -14/UNAUTHORIZED
(9)[   498571.483]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498571.483]    WARN ziti-sdk:bind.c:210 session_cb() server[0.50] failed to get session for service[tcp-10.10.10.55]: -14/UNAUTHORIZED
(9)[   498573.728]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498573.728]    WARN ziti-sdk:bind.c:210 session_cb() server[0.76] failed to get session for service[HOST-tcp-10.10.10.23-6443-INTERCEPTS-tcp-10.10.10.23-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498575.310]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498575.310]    WARN ziti-sdk:bind.c:210 session_cb() server[0.96] failed to get session for service[HOST-tcp-10.10.10.54-6443-INTERCEPTS-tcp-10.10.10.54-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498577.490]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498577.490]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
(9)[   498579.145]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498579.145]    WARN ziti-sdk:bind.c:210 session_cb() server[0.96] failed to get session for service[HOST-tcp-10.10.10.54-6443-INTERCEPTS-tcp-10.10.10.54-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498584.084]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498584.084]    WARN ziti-sdk:bind.c:210 session_cb() server[0.3] failed to get session for service[patras-svc-gw]: -14/UNAUTHORIZED
(9)[   498584.862]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498584.862]    WARN ziti-sdk:bind.c:210 session_cb() server[0.44] failed to get session for service[HOST-tcp-10.10.10.55-6443-INTERCEPTS-tcp-10.10.10.55-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498585.651]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(9)[   498585.651]    WARN ziti-sdk:ziti.c:1598 api_session_cb() ztx[0] failed to get api session from ctrl[https://ztc.euprojects.net:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection timed out
(9)[   498585.981]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498585.981]    WARN ziti-sdk:bind.c:210 session_cb() server[0.71] failed to get session for service[HOST-tcp-10.10.10.52-6443-INTERCEPTS-tcp-10.10.10.52-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498586.072]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498586.072]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
(9)[   498588.666]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498588.666]    WARN ziti-sdk:bind.c:210 session_cb() server[0.76] failed to get session for service[HOST-tcp-10.10.10.23-6443-INTERCEPTS-tcp-10.10.10.23-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498589.138]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498589.138]    WARN ziti-sdk:bind.c:210 session_cb() server[0.96] failed to get session for service[HOST-tcp-10.10.10.54-6443-INTERCEPTS-tcp-10.10.10.54-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498589.418]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498589.418]    WARN ziti-sdk:bind.c:210 session_cb() server[0.71] failed to get session for service[HOST-tcp-10.10.10.52-6443-INTERCEPTS-tcp-10.10.10.52-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498590.651]    INFO ziti-sdk:ziti.c:914 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://ztc.euprojects.net:8441] api_session_status[0] api_session_expired[TRUE]
(9)[   498592.787]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498592.787]    WARN ziti-sdk:bind.c:210 session_cb() server[0.62] failed to get session for service[HOST-tcp-10.10.10.19-6443-INTERCEPTS-tcp-10.10.10.19-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498596.181]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498596.181]    WARN ziti-sdk:bind.c:210 session_cb() server[0.62] failed to get session for service[HOST-tcp-10.10.10.19-6443-INTERCEPTS-tcp-10.10.10.19-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498596.374]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498596.374]    WARN ziti-sdk:bind.c:210 session_cb() server[0.96] failed to get session for service[HOST-tcp-10.10.10.54-6443-INTERCEPTS-tcp-10.10.10.54-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498596.596]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498596.596]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
(9)[   498597.103]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498597.103]    WARN ziti-sdk:bind.c:210 session_cb() server[0.5] failed to get session for service[internal-svc-k8s]: -14/UNAUTHORIZED
(9)[   498599.817]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498599.817]    WARN ziti-sdk:bind.c:210 session_cb() server[0.3] failed to get session for service[patras-svc-gw]: -14/UNAUTHORIZED
(9)[   498600.392]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498600.392]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
(9)[   498600.969]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498600.969]    WARN ziti-sdk:bind.c:210 session_cb() server[0.50] failed to get session for service[tcp-10.10.10.55]: -14/UNAUTHORIZED
(9)[   498603.425]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498603.425]    WARN ziti-sdk:bind.c:210 session_cb() server[0.76] failed to get session for service[HOST-tcp-10.10.10.23-6443-INTERCEPTS-tcp-10.10.10.23-6443-TO-6443]: -14/UNAUTHORIZED
(9)[   498603.784]    WARN ziti-sdk:ziti_ctrl.c:487 verify_api_session() ctrl[ztc.euprojects.net] no API session
(9)[   498603.784]    WARN ziti-sdk:bind.c:210 session_cb() server[0.22] failed to get session for service[tcp-10.10.10.61]: -14/UNAUTHORIZED
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Naturally we can use the restart: always option to mitigate this issue, but we would like to know the cause of the problem. I have also added the following logging object in a recently updated docker-compose.yaml file:

logging:
    driver: "json-file"
    options:
        max-size: "10m"
        max-file: "3"

While this will help reduce the storage size of the ziti container logs (which I assume it is natural for them to fill up after some time), I am not entirely certain if this is the root issue. It is of course a bit difficult to replicate the error, since the exits appear to be quite random. In fact, some machines are rarely, if ever affected.

Any assistance would be greatly appreciated.

qrkourier · September 19, 2024, 12:56pm

Hello! Welcome to the OpenZiti community.

Correct, the tunneler is not expected to exit in the way you observed. It should run continuously even if no identities are added.

The tunneler presents an IPC socket for managing identities, and I have found it somewhat more intuitive to interact with it as a Linux service than as a Docker container, especially for debugging issues like this!

Still, there's no reason why it shouldn't work as a Docker container. The K8S node proxy daemonset uses the same container image with a similar configuration as you have shared.

As we proceed to diagnose this issue, let's establish the precise version of the tunneler you're using and please consider switching to the Linux tunneler service to reproduce the issue so we'll have an easier time collecting environment info with the script included in the Linux package: /opt/openziti/bin/debug.bash.

The most direct way to gather more information from the Docker tunneler is to enable verbose (DEBUG) log level.

Here, I have edited your compose.yml file to add the --verbose=4, set the tunneler version to the latest stable release, 1.1.3, and set an env vars to increase library log verbosity as well.

services:
    ziti-tun:
        image: docker.io/openziti/ziti-edge-tunnel:1.1.3
        devices:
            - /dev/net/tun:/dev/net/tun
        volumes:
            - .:/ziti-edge-tunnel
            - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
        environment:
            - ZITI_IDENTITY_BASENAME=ziti_id
            - PFXLOG_NO_JSON=true              # suppress JSON logging
            - TLSUV_DEBUG=4
            - ZITI_LOG=4
        network_mode: host
        privileged: true
        command: run --verbose=4

options, modes, and troubleshooting docs: Options and Modes | OpenZiti

pantmal · October 8, 2024, 2:58pm

Hello @qrkourier

We've just had a new abrupt exit on our ziti edge tunnel container, using the verbose option you recommended, as well as the latest stable release of 1.1.3

I am attaching the logs of the exited container. Naturally, there were a lot of lines, as well as repeating ones, so I saved the last 100 lines. I have saved up to the last 1000 lines, so let me know if you want see additional lines too.

root@nsit-cd-openslice:/home/ubuntu/ziti# docker logs -f 3660b8d68d51 --tail 100
(7)[  1211500.027]   DEBUG ziti-sdk:bind.c:172 schedule_rebind() server[0.2] scheduling re-bind in 300.0s
(7)[  1211506.855]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211507.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211507.105]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.250 s
(7)[  1211507.105]   DEBUG ziti-sdk:ziti_ctrl.c:351 ctrl_body_cb() ctrl[ztc.euprojects.net] received 1/1 for paging request GET[/current-identity/edge-routers]
(7)[  1211507.105]   DEBUG ziti-sdk:ziti_ctrl.c:363 ctrl_body_cb() ctrl[ztc.euprojects.net] completed paging request GET[/current-identity/edge-routers] in 0.250 s
(7)[  1211507.149]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-api-session/service-updates] in 0.294 s
(7)[  1211517.149]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211517.386]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.236 s
(7)[  1211517.386]   DEBUG ziti-sdk:ziti_ctrl.c:351 ctrl_body_cb() ctrl[ztc.euprojects.net] received 1/1 for paging request GET[/current-identity/edge-routers]
(7)[  1211517.386]   DEBUG ziti-sdk:ziti_ctrl.c:363 ctrl_body_cb() ctrl[ztc.euprojects.net] completed paging request GET[/current-identity/edge-routers] in 0.236 s
(7)[  1211517.431]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-api-session/service-updates] in 0.281 s
(7)[  1211527.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211527.431]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211527.690]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.259 s
(7)[  1211527.690]   DEBUG ziti-sdk:ziti_ctrl.c:351 ctrl_body_cb() ctrl[ztc.euprojects.net] received 1/1 for paging request GET[/current-identity/edge-routers]
(7)[  1211527.690]   DEBUG ziti-sdk:ziti_ctrl.c:363 ctrl_body_cb() ctrl[ztc.euprojects.net] completed paging request GET[/current-identity/edge-routers] in 0.259 s
(7)[  1211527.733]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-api-session/service-updates] in 0.302 s
(7)[  1211537.733]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211537.948]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.214 s
(7)[  1211537.948]   DEBUG ziti-sdk:ziti_ctrl.c:351 ctrl_body_cb() ctrl[ztc.euprojects.net] received 1/1 for paging request GET[/current-identity/edge-routers]
(7)[  1211537.948]   DEBUG ziti-sdk:ziti_ctrl.c:363 ctrl_body_cb() ctrl[ztc.euprojects.net] completed paging request GET[/current-identity/edge-routers] in 0.215 s
(7)[  1211537.988]   DEBUG ziti-sdk:ziti_ctrl.c:334 ctrl_body_cb() ctrl[ztc.euprojects.net] completed GET[/current-api-session/service-updates] in 0.254 s
(7)[  1211547.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211547.988]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211562.988]   DEBUG tlsuv:http.c:289 failed to connect: -110(connection timed out)
(7)[  1211562.988]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(7)[  1211562.988]   ERROR ziti-sdk:ziti.c:1317 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/connection timed out
(7)[  1211562.988]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(7)[  1211562.988]    WARN ziti-sdk:ziti.c:1262 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/connection timed out]
(7)[  1211567.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211572.988]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211587.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211587.988]   DEBUG tlsuv:http.c:289 failed to connect: -110(connection timed out)
(7)[  1211587.988]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(7)[  1211587.988]   ERROR ziti-sdk:ziti.c:1317 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/connection timed out
(7)[  1211587.988]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -110(connection timed out)
(7)[  1211587.988]    WARN ziti-sdk:ziti.c:1262 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/connection timed out]
(7)[  1211588.029]   DEBUG tlsuv:http.c:289 failed to connect: -3008(unknown node or service)
(7)[  1211594.520]   ERROR ziti-sdk:channel.c:620 latency_timeout() ch[0] no read/write traffic on channel since before latency probe was sent, closing channel
(7)[  1211594.520]   DEBUG ziti-sdk:channel.c:90 close_connection() ch[0] closing TLS[0x55e044297890]
(7)[  1211594.520]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:846 on_ziti_event() ztx[ubi-openslice-192.168.5.160] router ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router disconnected
(7)[  1211594.520]    WARN ziti-sdk:bind.c:376 on_message() binding failed: -20/operation did not complete in time
(7)[  1211594.520]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.2] scheduling re-bind(attempt=1) in 0.978s
(7)[  1211594.520]    WARN ziti-sdk:bind.c:376 on_message() binding failed: -20/operation did not complete in time
(7)[  1211594.520]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.0] scheduling re-bind(attempt=1) in 0.189s
(7)[  1211594.520]    WARN ziti-sdk:bind.c:376 on_message() binding failed: -20/operation did not complete in time
(7)[  1211594.520]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.1] scheduling re-bind(attempt=1) in 0.493s
(7)[  1211594.520]    INFO ziti-sdk:channel.c:775 reconnect_channel() ch[0] reconnecting in 2087ms (attempt = 1)
(7)[  1211594.709]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.0] staring re-bind
(7)[  1211595.013]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.1] staring re-bind
(7)[  1211595.498]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.2] staring re-bind
(7)[  1211596.607]   DEBUG ziti-sdk:channel.c:743 reconnect_cb() ch[0] connecting to tls://ztc.euprojects.net:8442
(7)[  1211597.988]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211607.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211613.029]   DEBUG tlsuv:http.c:289 failed to connect: -3008(unknown node or service)
(7)[  1211613.029]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211613.029]    WARN ziti-sdk:bind.c:214 session_cb() server[0.0] failed to get session for service[prod-ubi-openslice]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211613.029]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.0] scheduling re-bind(attempt=2) in 0.878s
(7)[  1211613.029]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211613.029]    WARN ziti-sdk:bind.c:214 session_cb() server[0.1] failed to get session for service[prod-internal-k8s-213]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211613.029]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.1] scheduling re-bind(attempt=2) in 0.859s
(7)[  1211613.029]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211613.029]    WARN ziti-sdk:bind.c:214 session_cb() server[0.2] failed to get session for service[prod-internal-prometheus-213]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211613.029]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.2] scheduling re-bind(attempt=2) in 1.446s
(7)[  1211613.029]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211613.029]   ERROR ziti-sdk:ziti.c:1317 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/unknown node or service
(7)[  1211613.029]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211613.029]    WARN ziti-sdk:ziti.c:1262 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/unknown node or service]
(7)[  1211613.888]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.1] staring re-bind
(7)[  1211613.907]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.0] staring re-bind
(7)[  1211614.475]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.2] staring re-bind
(7)[  1211616.607]   ERROR ziti-sdk:channel.c:709 ch_connect_timeout() ch[0] connect timeout
(7)[  1211616.607]    INFO ziti-sdk:channel.c:775 reconnect_channel() ch[0] reconnecting in 6718ms (attempt = 2)
(7)[  1211616.607]   DEBUG ziti-sdk:channel.c:90 close_connection() ch[0] closing TLS[0x55e0442cdb20]
(7)[  1211616.607]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router] [-125/operation canceled]
(7)[  1211623.030]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211623.325]   DEBUG ziti-sdk:channel.c:743 reconnect_cb() ch[0] connecting to tls://ztc.euprojects.net:8442
(7)[  1211627.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211634.750]   DEBUG tlsuv:http.c:289 failed to connect: -3008(unknown node or service)
(7)[  1211634.750]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211634.750]    WARN ziti-sdk:bind.c:214 session_cb() server[0.1] failed to get session for service[prod-internal-k8s-213]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211634.750]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.1] scheduling re-bind(attempt=3) in 2.929s
(7)[  1211634.750]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211634.750]    WARN ziti-sdk:bind.c:214 session_cb() server[0.0] failed to get session for service[prod-ubi-openslice]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211634.750]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.0] scheduling re-bind(attempt=3) in 3.683s
(7)[  1211634.750]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211634.750]    WARN ziti-sdk:bind.c:214 session_cb() server[0.2] failed to get session for service[prod-internal-prometheus-213]: -16/CONTROLLER_UNAVAILABLE
(7)[  1211634.750]   DEBUG ziti-sdk:bind.c:167 schedule_rebind() server[0.2] scheduling re-bind(attempt=3) in 3.504s
(7)[  1211634.750]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211634.750]   ERROR ziti-sdk:ziti.c:1317 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/unknown node or service
(7)[  1211634.750]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ztc.euprojects.net] request failed: -3008(unknown node or service)
(7)[  1211634.750]    WARN ziti-sdk:ziti.c:1262 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/unknown node or service]
(7)[  1211637.679]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.1] staring re-bind
(7)[  1211638.254]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.2] staring re-bind
(7)[  1211638.433]   DEBUG ziti-sdk:bind.c:97 rebind_delay_cb() server[0.0] staring re-bind
(7)[  1211643.325]   ERROR ziti-sdk:channel.c:709 ch_connect_timeout() ch[0] connect timeout
(7)[  1211643.325]    INFO ziti-sdk:channel.c:775 reconnect_channel() ch[0] reconnecting in 3881ms (attempt = 3)
(7)[  1211643.325]   DEBUG ziti-sdk:channel.c:90 close_connection() ch[0] closing TLS[0x55e0442cdb20]
(7)[  1211643.325]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router] [-125/operation canceled]
(7)[  1211644.750]   DEBUG ziti-sdk:ziti_ctrl.c:726 ctrl_paging_req() ctrl[ztc.euprojects.net] starting paging request GET[/current-identity/edge-routers]
(7)[  1211647.019]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
(7)[  1211647.206]   DEBUG ziti-sdk:channel.c:743 reconnect_cb() ch[0] connecting to tls://ztc.euprojects.net:8442
/docker-entrypoint.sh: line 184:     7 Segmentation fault      (core dumped) ziti-edge-tunnel "${TUNNEL_RUN_MODE}" "${TUNNEL_OPTS[@]}" "${@}"

The last line indicates a segmentation fault, although I am uncertain as to what is the cause of it.

Would like to hear your thoughts on this.

qrkourier · October 8, 2024, 3:19pm

OK, we'll need the core file to diagnose. Was it still running in a container? If so, there may be additional setup steps to capture the crash dump.

Link to linux crash troubleshooting doc

scareything · October 8, 2024, 3:33pm

I also suggest getting 1.1.5, since several fixes have been made, and setting the verbosity to "6" (TRACE).

pantmal · October 22, 2024, 12:10pm

Hello we have tried setting the verbosity to '6'. I am attaching the last 200 logs of the last exited container:

(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247873] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.356730/75zkG29w/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247874] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247874] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247874] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.356725/YbZgwJbn/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247875] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247875] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247875] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.419321/uR9IHGO9/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247876] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247876] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247876] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.389447/QzbJSzNX/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247877] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247877] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247877] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.395163/L1YvDwMh/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247878] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247878] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247878] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.356731/-t7ahyf5/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247879] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247879] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247879] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.395154/xumKRl-G/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247880] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247880] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247880] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.25300/XIuUwFzz/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247881] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247881] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247881] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.386366/dQ6zdeDc/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247882] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247882] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247882] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.419322/rdSUVpGU/Connected] starting flusher
(7)[   179586.524]   TRACE ziti-sdk:channel.c:558 process_inbound() ch[0] <= ct[ED71] seq[4247883] len[12] hdrs[24]
(7)[   179586.524]   TRACE ziti-sdk:channel.c:569 process_inbound() ch[0] completing msg seq[4247883] body+hrds=12+24, in_offset=0, want=36, got=36
(7)[   179586.524]   TRACE ziti-sdk:channel.c:580 process_inbound() ch[0] message is complete seq[4247883] ct[ED71]
(7)[   179586.524]   TRACE ziti-sdk:connect.c:764 flush_connection() conn[0.47991/7s9H3Rju/Connected] starting flusher
(7)[   179586.524]   TRACE tlsuv:tlsuv.c:415 finished reading after 1 iterations
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.47991/7s9H3Rju/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.47991/7s9H3Rju/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.47991/7s9H3Rju/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.47991/7s9H3Rju/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.47991/7s9H3Rju/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.47991/7s9H3Rju/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.47991/7s9H3Rju/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.419322/rdSUVpGU/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.419322/rdSUVpGU/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419322/rdSUVpGU/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.419322/rdSUVpGU/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419322/rdSUVpGU/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.419322/rdSUVpGU/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.419322/rdSUVpGU/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.386366/dQ6zdeDc/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.386366/dQ6zdeDc/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.386366/dQ6zdeDc/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.386366/dQ6zdeDc/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.386366/dQ6zdeDc/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.386366/dQ6zdeDc/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.386366/dQ6zdeDc/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.25300/XIuUwFzz/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.25300/XIuUwFzz/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25300/XIuUwFzz/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.25300/XIuUwFzz/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25300/XIuUwFzz/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.25300/XIuUwFzz/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.25300/XIuUwFzz/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.395154/xumKRl-G/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.395154/xumKRl-G/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395154/xumKRl-G/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.395154/xumKRl-G/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395154/xumKRl-G/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.395154/xumKRl-G/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.395154/xumKRl-G/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.356731/-t7ahyf5/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.356731/-t7ahyf5/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356731/-t7ahyf5/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.356731/-t7ahyf5/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356731/-t7ahyf5/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.356731/-t7ahyf5/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.356731/-t7ahyf5/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.395163/L1YvDwMh/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.395163/L1YvDwMh/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395163/L1YvDwMh/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.395163/L1YvDwMh/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395163/L1YvDwMh/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.395163/L1YvDwMh/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.395163/L1YvDwMh/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.389447/QzbJSzNX/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.389447/QzbJSzNX/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.389447/QzbJSzNX/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.389447/QzbJSzNX/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.389447/QzbJSzNX/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.389447/QzbJSzNX/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.389447/QzbJSzNX/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.419321/uR9IHGO9/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.419321/uR9IHGO9/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419321/uR9IHGO9/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.419321/uR9IHGO9/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419321/uR9IHGO9/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.419321/uR9IHGO9/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.419321/uR9IHGO9/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.356725/YbZgwJbn/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.356725/YbZgwJbn/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356725/YbZgwJbn/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.356725/YbZgwJbn/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356725/YbZgwJbn/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.356725/YbZgwJbn/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.356725/YbZgwJbn/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.356730/75zkG29w/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.356730/75zkG29w/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356730/75zkG29w/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.356730/75zkG29w/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.356730/75zkG29w/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.356730/75zkG29w/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.356730/75zkG29w/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.25299/SpYlr1hX/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.25299/SpYlr1hX/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25299/SpYlr1hX/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.25299/SpYlr1hX/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25299/SpYlr1hX/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.25299/SpYlr1hX/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.25299/SpYlr1hX/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.25297/nBEEYzRS/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.25297/nBEEYzRS/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25297/nBEEYzRS/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.25297/nBEEYzRS/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.25297/nBEEYzRS/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.25297/nBEEYzRS/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.25297/nBEEYzRS/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.419275/sciej1HA/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.419275/sciej1HA/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419275/sciej1HA/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.419275/sciej1HA/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419275/sciej1HA/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.419275/sciej1HA/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.419275/sciej1HA/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.419263/IoWoTFEE/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.419263/IoWoTFEE/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419263/IoWoTFEE/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.419263/IoWoTFEE/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.419263/IoWoTFEE/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.419263/IoWoTFEE/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.419263/IoWoTFEE/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.395164/b2qBEb7B/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.395164/b2qBEb7B/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395164/b2qBEb7B/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.395164/b2qBEb7B/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.395164/b2qBEb7B/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.395164/b2qBEb7B/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.395164/b2qBEb7B/Closed] stopping flusher
(7)[   179586.525]   TRACE ziti-sdk:connect.c:1330 process_edge_message() conn[0.406121/WG6q291D/Connected] <= ct[ED71] edge_seq[4] len[12]
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:1336 process_edge_message() conn[0.406121/WG6q291D/Connected] connection status[ED71] seq[4] err[close called]
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.406121/WG6q291D/Connected] transitioning Connected => Disconnected
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:811 flush_to_client() conn[0.406121/WG6q291D/Disconnected] 0 bytes available
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:111 on_ziti_data() got -24 bytes from ziti
(7)[   179586.525]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:137 on_ziti_data() ziti connection is closed due to [-24](connection is closed)
(7)[   179586.525] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.406121/WG6q291D/Disconnected] transitioning Disconnected => Closed
(7)[   179586.525]   TRACE ziti-sdk:connect.c:798 flush_to_service() conn[0.406121/WG6q291D/Closed] flushed 0 messages
(7)[   179586.525]   TRACE ziti-sdk:connect.c:757 on_flush() conn[0.406121/WG6q291D/Closed] stopping flusher
(7)[   179586.525]   DEBUG ziti-sdk:connect.c:172 close_conn_internal() conn[0.419322/rdSUVpGU/Closed] removing
(7)[   179586.525]   TRACE tunnel-cbs:ziti_tunnel_cbs.c:594 ziti_conn_close_cb() ziti_conn[0x561aa229d4c0] is closed
(7)[   179586.525]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[tcp:100.64.0.1:42689] service[tcp-10.10.10.76-30201]
/docker-entrypoint.sh: line 184:     7 Segmentation fault      ziti-edge-tunnel "${TUNNEL_RUN_MODE}" "${TUNNEL_OPTS[@]}" "${@}"

This execution was performed with the 1.1.4 version (I didn't remember the mention of 1.1.5). Should this container exit again, I will try with version 1.1.5.

Now regarding the core dump, I discovered that the options in order to enable core dumps were disabled on my machine, so I will enable this option now, and come back again.

Topic		Replies	Views
Ziti Edge Router can't connect after some time (Docker Compose) Support	14	359	April 27, 2023
Android Tunneler connection errors Tunneler Apps	10	176	March 5, 2024
Crypto Failure in ziti-edge-tunnel Support	9	215	September 15, 2023
I think quickstart:0.30.1 broke something in the edge-tunnel connections	18	406	September 1, 2023
How to use zitiLogin inside Ziti Edge Controller Docker container General Questions	6	451	April 17, 2023

Ziti Edge Tunnel Docker container fails/exits

Related Topics