Hi there,
we have a problem that I believe is not solely caused by Ziti itself but crashes the tunneler eventually:
Feb 15 10:07:11 mss systemd[1]: Starting Ziti Edge Tunnel...
Feb 15 10:07:11 mss ziti-edge-tunnel.sh[25891]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Feb 15 10:07:11 mss systemd[1]: Started Ziti Edge Tunnel.
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.15-local)
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO tunnel-cbs:ziti_dns.c:171 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1698 run_tunneler_loop() Loading identity files from /opt/openziti/etc/identities
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1102 load_identities() loading identity file: gl01.json
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1624 make_socket_path() effective group set to 'ziti' (gid=996)
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.010] INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.010] INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/opt/openziti/etc/identities/gl01.json]
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.010] INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/opt/openziti/etc/identities/gl01.json]
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.010] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1117 load_id_cb() identity[/opt/openziti/etc/identities/gl01.json] loaded
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.012] INFO ziti-sdk:ziti.c:449 ziti_init_async() ztx[0] using tlsuv[v0.27.0], tls[mbed TLS 2.28.1]
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.012] INFO ziti-sdk:ziti.c:450 ziti_init_async() ztx[0] Loading ziti context with controller[https://zt.company.de:8441]
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.012] INFO ziti-sdk:ziti.c:914 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://zt.company.de:8441] api_session_status[0] api_session_expired[TRUE]
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.012] INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.116] INFO ziti-sdk:ziti.c:1672 version_cb() ztx[0] connected to controller https://zt.company.de:8441 version v0.31.4(1c21434737ac 2023-12-18T16:24:57Z)
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.125] ERROR ziti-sdk:ziti.c:1534 ziti_set_api_session() ztx[0] local clock is 40 seconds ahead UTC (as reported by controller)
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.125] INFO ziti-sdk:ziti.c:1561 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
Feb 15 10:07:11 mss ziti-edge-tunnel[25892]: (25892)[ 0.125] INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[gl01.company.ziti] connected to controller
Feb 15 10:07:26 mss ziti-edge-tunnel[25892]: (25892)[ 0.125] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/opt/openziti/etc/identities/gl01.json] context event : status is OK
Feb 15 10:07:26 mss ziti-edge-tunnel[25892]: (25892)[ 15.176] INFO ziti-sdk:posture.c:202 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti.c:1365 update_identity_data() ztx[0] failed to get identity_data: connection timed out[CONTROLLER_UNAVAILABLE]
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] WARN tunnel-cbs:ziti_tunnel_ctrl.c:781 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/opt/openziti/etc/identities/gl01.json] context event : status is ziti controller is not available
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1202 on_event() ztx[/opt/openziti/etc/identities/gl01.json] failed to connect to controller due to ziti controller is not available
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti.c:1452 ca_bundle_cb() failed to get CA bundle from controller: connection timed out
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti.c:1365 update_identity_data() ztx[0] failed to get identity_data: connection timed out[CONTROLLER_UNAVAILABLE]
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:07:41 mss ziti-edge-tunnel[25892]: (25892)[ 30.176] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:08:06 mss ziti-edge-tunnel[25892]: (25892)[ 55.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:06 mss ziti-edge-tunnel[25892]: (25892)[ 55.176] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:08:06 mss ziti-edge-tunnel[25892]: (25892)[ 55.176] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:06 mss ziti-edge-tunnel[25892]: (25892)[ 55.176] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:08:31 mss ziti-edge-tunnel[25892]: (25892)[ 80.177] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:31 mss ziti-edge-tunnel[25892]: (25892)[ 80.177] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:08:31 mss ziti-edge-tunnel[25892]: (25892)[ 80.177] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:31 mss ziti-edge-tunnel[25892]: (25892)[ 80.177] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:08:56 mss ziti-edge-tunnel[25892]: (25892)[ 105.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:56 mss ziti-edge-tunnel[25892]: (25892)[ 105.178] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:08:56 mss ziti-edge-tunnel[25892]: (25892)[ 105.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:08:56 mss ziti-edge-tunnel[25892]: (25892)[ 105.178] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:09:21 mss ziti-edge-tunnel[25892]: (25892)[ 130.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:09:21 mss ziti-edge-tunnel[25892]: (25892)[ 130.178] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:09:21 mss ziti-edge-tunnel[25892]: (25892)[ 130.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:09:21 mss ziti-edge-tunnel[25892]: (25892)[ 130.178] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:09:46 mss ziti-edge-tunnel[25892]: (25892)[ 155.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:09:46 mss ziti-edge-tunnel[25892]: (25892)[ 155.178] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:09:46 mss ziti-edge-tunnel[25892]: (25892)[ 155.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:09:46 mss ziti-edge-tunnel[25892]: (25892)[ 155.178] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:10:11 mss ziti-edge-tunnel[25892]: (25892)[ 180.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:10:11 mss ziti-edge-tunnel[25892]: (25892)[ 180.178] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:10:11 mss ziti-edge-tunnel[25892]: (25892)[ 180.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:10:11 mss ziti-edge-tunnel[25892]: (25892)[ 180.178] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:10:36 mss ziti-edge-tunnel[25892]: (25892)[ 205.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:10:36 mss ziti-edge-tunnel[25892]: (25892)[ 205.178] ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
Feb 15 10:10:36 mss ziti-edge-tunnel[25892]: (25892)[ 205.178] ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -110(connection timed out)
Feb 15 10:10:36 mss ziti-edge-tunnel[25892]: (25892)[ 205.178] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://zt.company.de:8441]
Feb 15 10:10:51 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO ziti-sdk:channel.c:247 new_ziti_channel() ch[0] (zt-router-1.company.ziti) new channel for ztx[0] identity[gl01.company.ziti]
Feb 15 10:10:51 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[gl01.company.ziti] added edge router zt-router-1.company.ziti@zt-router-1.company.de
Feb 15 10:10:56 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO ziti-sdk:channel.c:749 reconnect_channel() ch[0] reconnecting NOW
Feb 15 10:10:56 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO ziti-sdk:channel.c:247 new_ziti_channel() ch[1] (zt-edge-router) new channel for ztx[0] identity[gl01.company.ziti]
Feb 15 10:10:56 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[gl01.company.ziti] added edge router zt-edge-router@zt.company.de
Feb 15 10:11:01 mss ziti-edge-tunnel[25892]: (25892)[ 220.225] INFO ziti-sdk:channel.c:749 reconnect_channel() ch[1] reconnecting NOW
Feb 15 10:11:06 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[gl01.company.ziti] connected to controller
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/opt/openziti/etc/identities/gl01.json] context event : status is OK
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:127.0.0.1:22] service[wildcard.ssh]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[gl_alert_to_soar.svc] with intercept.v1 = {"addresses":["soar.company.ziti"],"dialOptions":{"connectTimeoutSeconds":5,"identity":""},"portRanges":[{"high":5000,"low":5000}],"protocols":["tcp"],"sourceIp":""}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry soar.company.ziti -> 100.64.0.3
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[gl_alert_to_soar.svc]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:127.0.0.1:9000] service[gl_client_access.svc]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zabbix_agent.svc] with intercept.v1 = {"addresses":["zabbix.company.ziti"],"portRanges":[{"high":10051,"low":10051}],"protocols":["tcp"]}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry zabbix.company.ziti -> 100.64.0.4
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zabbix_agent.svc]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[company_pull_kafka_logs_bastion] with ziti-tunneler-client.v1 = {"hostname":"bastion.company.de","port":9092}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry bastion.company.de -> 100.64.0.5
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[company_pull_kafka_logs_bastion]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:127.0.0.1:5001] service[company_Send_Controller_Logs_to_GL01]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[udp:127.0.0.1:5000] service[lc.svc]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:127.0.0.1:3000] service[mn01_client_access.svc]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[LFT_pull_kafka_logs] with ziti-tunneler-client.v1 = {"hostname":"lft_appliance.company.ziti","port":9092}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry lft_appliance.company.ziti -> 100.64.0.6
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[LFT_pull_kafka_logs]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[company_Pull_Case_Metrics_From_Soar] with ziti-tunneler-client.v1 = {"hostname":"soar.company.ziti","port":80}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[company_Pull_Case_Metrics_From_Soar]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[EVB_pull_kafka_logs_EVB] with ziti-tunneler-client.v1 = {"hostname":"evbappliance.company.de","port":9092}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry evbappliance.company.de -> 100.64.0.7
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[EVB_pull_kafka_logs_EVB]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[company_Allow_GL01_to_EL01_9200] with ziti-tunneler-client.v1 = {"hostname":"el01.company.de","port":9200}
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry el01.company.de -> 100.64.0.8
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[company_Allow_GL01_to_EL01_9200]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - wildcard.ssh:hdr427wpfKVRPS0gX5BVO ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - gl_alert_to_soar.svc:fDHUs2T9FGKxTdLWioVSc ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - gl_client_access.svc:7moMQoJKRqz9o9COul31TN ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - zabbix_agent.svc:77tfbDtj89PVPirOcMCvjy ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - company_pull_kafka_logs_bastion:5zRnSSNesjj8Bia6i08SLB ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - company_Send_Controller_Logs_to_GL01:5BmIeNcBHmheOV5ivJPqTd ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - lc.svc:4GREROoH2qkzmW3YzT13w1 ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - mn01_client_access.svc:49V3t6a8JfkZMH4TYB7raZ ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - LFT_pull_kafka_logs:35HMBsrAn5RSuPuRAgTVOr ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - company_Pull_Case_Metrics_From_Soar:2E6OZDxCMX4H01DSB6tvxp ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - EVB_pull_kafka_logs_EVB:1xA7Ihh6NTc4gGus43VYJQ ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - company_Allow_GL01_to_EL01_9200:1BomK42mN1X7qqGaoemXUC ===============
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 235.367] INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 7 route updates
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.921] INFO ziti-sdk:channel.c:838 on_channel_data() ch[1] channel was closed [-103/software caused connection abort]
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.921] ERROR ziti-sdk:channel.c:637 hello_reply_cb() ch[1] failed to receive Hello response due to -21(connection to edge router terminated)
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.921] INFO tunnel-cbs:ziti_tunnel_ctrl.c:852 on_ziti_event() ztx[gl01.company.ziti] router zt-edge-router is unavailable
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.921] INFO ziti-sdk:channel.c:746 reconnect_channel() ch[1] reconnecting in 3597ms (attempt = 1)
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.921] INFO ziti-sdk:channel.c:746 reconnect_channel() ch[1] reconnecting in 18612ms (attempt = 2)
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.923] INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[7]: 0/OK
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.924] INFO ziti-sdk:channel.c:645 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.31.4|1c21434737ac|2023-12-18T16:24:57Z|linux|amd64
Feb 15 10:11:08 mss ziti-edge-tunnel[25892]: (25892)[ 236.924] INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[gl01.company.ziti] router zt-router-1.company.ziti connected
Feb 15 10:11:28 mss ziti-edge-tunnel[25892]: (25892)[ 256.994] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.4] for query[1:zabbix.company.ziti]
Feb 15 10:11:28 mss ziti-edge-tunnel[25892]: (25892)[ 257.075] INFO ziti-sdk:channel.c:645 hello_reply_cb() ch[1] connected. EdgeRouter version: v0.31.4|1c21434737ac|2023-12-18T16:24:57Z|linux|amd64
Feb 15 10:11:28 mss ziti-edge-tunnel[25892]: (25892)[ 257.075] INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[gl01.company.ziti] router zt-edge-router connected
Feb 15 10:11:38 mss ziti-edge-tunnel[25892]: (25892)[ 266.940] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.6] for query[1:lft_appliance.company.ziti]
Feb 15 10:11:38 mss ziti-edge-tunnel[25892]: (25892)[ 267.150] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.5] for query[1:bastion.company.de]
Feb 15 10:11:38 mss ziti-edge-tunnel[25892]: (25892)[ 267.303] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.7] for query[1:evbappliance.company.de]
Feb 15 10:11:48 mss ziti-edge-tunnel[25892]: (25892)[ 277.833] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[company_Send_Controller_Logs_to_GL01] client[zt.company.ziti] client_src_addr[tcp:100.64.0.1:52500] dst_addr[tcp:127.0.0.1:5001]: incoming connection
Feb 15 10:12:22 mss ziti-edge-tunnel[25892]: (25892)[ 311.154] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[wildcard.ssh] client[dm_mb] client_src_addr[tcp:100.64.0.1:54601] dst_addr[tcp:127.0.0.1:22]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.018] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54604] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.273] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54605] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.278] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54606] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.279] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54609] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.279] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54607] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.279] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54608] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] ERROR ziti-sdk:channel.c:844 on_channel_data() ch[0] unhandled error on_data [-9/bad file descriptor]
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] INFO tunnel-cbs:ziti_tunnel_ctrl.c:846 on_ziti_event() ztx[gl01.company.ziti] router zt-router-1.company.ziti disconnected
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.54] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.53] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.52] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.51] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.50] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.49] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.48] closing bridge due to error: -21(connection to edge router terminated)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] WARN ziti-sdk:bind.c:346 on_message() binding failed: -21/connection to edge router terminated
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.413] INFO ziti-sdk:channel.c:746 reconnect_channel() ch[0] reconnecting in 2348ms (attempt = 1)
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.493] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54610] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.493] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54611] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.519] INFO ziti-sdk:channel.c:749 reconnect_channel() ch[0] reconnecting NOW
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 5.138 q=31 qs=959300
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] ERROR ziti-sdk:channel.c:357 on_channel_send() ch[0] write failed [-103/software caused connection abort]
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] ERROR ziti-sdk:channel.c:637 hello_reply_cb() ch[0] failed to receive Hello response due to -24(connection is closed)
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] INFO tunnel-cbs:ziti_tunnel_ctrl.c:852 on_ziti_event() ztx[gl01.company.ziti] router zt-router-1.company.ziti is unavailable
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] INFO ziti-sdk:channel.c:746 reconnect_channel() ch[0] reconnecting in 3252ms (attempt = 1)
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] INFO ziti-sdk:channel.c:746 reconnect_channel() ch[0] reconnecting in 19115ms (attempt = 2)
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] ERROR ziti-sdk:channel.c:357 on_channel_send() ch[0] write failed [-103/software caused connection abort]
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] ERROR ziti-sdk:channel.c:357 on_channel_send() ch[0] write failed [-103/software caused connection abort]
Feb 15 10:12:31 mss ziti-edge-tunnel[25892]: (25892)[ 320.530] ERROR ziti-sdk:channel.c:357 on_channel_send() ch[0] write failed [-103/software caused connection abort]
Feb 15 10:12:31 mss systemd[1]: ziti-edge-tunnel.service: Main process exited, code=killed, status=11/SEGV
Feb 15 10:12:31 mss systemd[1]: ziti-edge-tunnel.service: Failed with result 'signal'.
Feb 15 10:12:34 mss systemd[1]: ziti-edge-tunnel.service: Service RestartSec=3s expired, scheduling restart.
Feb 15 10:12:34 mss systemd[1]: ziti-edge-tunnel.service: Scheduled restart job, restart counter is at 5.
Feb 15 10:12:34 mss systemd[1]: Stopped Ziti Edge Tunnel.
Feb 15 10:12:34 mss systemd[1]: Starting Ziti Edge Tunnel...
### Ziti-Edge-Tunnel restarts due to systemd restart policy
Interestingly enough the ziti-edge-tunnel is killed exactly when I try to access one server at port 9000 which is represented by the following log:
Feb 15 10:12:26 mss ziti-edge-tunnel[25892]: (25892)[ 315.018] INFO tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[gl_client_access.svc] client[dm_mb] client_src_addr[tcp:100.64.0.1:54604] dst_addr[tcp:127.0.0.1:9000]: incoming connection
Since we were running v0.22.15, could this be the crashing which v.22.18 fixed?