I was also leaning towards numerous connections, as when managing the current sessions in ZAC, I can see multiple sessions and API sessions for the same identity/service. Could this be due to the multiple ssh connections I have to the instance, even though only one of the them is running the tunneler. The Drone.EC2 service has two sessions created right after each other.
I ran another test today and was able to confirm connections using netstat locally on the instance.
You can see that on the instance, there are initially two connections for port 14550, but then when I send the message 'splendid' from my local machine, it does not reach the instance. Running netstat again shows that one of the connections dropped for some reason.
I am still running into the same issue of my tunneler not registering incoming connections for messages sent, even tho I was successfully communicating with the EC2 instance from my local machine using netcat.
I ran a status on the tunneler after and this is what was returned. Looks like all the messages I sent are actually logging -111/connection refused
systemctl status ziti-edge-tunnel
● ziti-edge-tunnel.service - Ziti Edge Tunnel
Loaded: loaded (/usr/lib/systemd/system/ziti-edge-tunnel.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-11-19 17:24:06 UTC; 22h ago
Main PID: 599 (ziti-edge-tunne)
Tasks: 6 (limit: 1130)
Memory: 12.7M (peak: 13.4M)
CPU: 23.441s
CGroup: /system.slice/ziti-edge-tunnel.service
└─599 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities
Nov 19 21:01:00 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 13013.422] WARN ziti-sdk:ziti.c:1801 ztx_auth_state_cb() auth error: The request could not be completed. The session is not authorized or the credentials are invalid
Nov 19 22:32:09 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18482.711] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.25] err = -111/connection refused
Nov 19 22:32:12 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18485.916] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.26] err = -111/connection refused
Nov 19 22:34:08 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18601.346] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.28] err = -111/connection refused
Nov 19 22:39:08 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18901.642] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.31] err = -111/connection refused
Nov 19 22:39:14 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18907.583] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.32] err = -111/connection refused
Nov 19 22:39:37 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 18930.799] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.33] err = -111/connection refused
Nov 20 15:21:28 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 79041.852] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.35] err = -111/connection refused
Nov 20 15:21:39 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 79052.641] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.36] err = -111/connection refused
Nov 20 15:22:07 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[ 79081.162] WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.37] err = -111/connection refused
Could the tunneler be having issues generating a valid handshake with the controller each time I ssh into the instance? Even though I can see valid API sessions as seen above. I tried another session and was able to talk from my laptop to the instance successfully. Here is what was returned for netstat and ip_dump. The tunneler is not still not logging incoming connections, but it did log my ip_dumps.
~$ netstat | grep 14550
udp 0 0 ip-172-31-1-229.e:14550 ec2-3-92-74-179.c:40772 ESTABLISHED
udp 0 0 ip-172-31-1-229.e:40772 ec2-3-92-74-179.c:14550 ESTABLISHED
ubuntu@ip-172-31-1-229:~$ sudo ziti-edge-tunnel ip_dump
{
"Success":true,
"Data":{
"Pools":[
{
"Name":"MEMP_PBUF_POOL",
"Max":1,
"Used":0,
"Avail":1024
},
{
"Name":"MEMP_TCP_PCB",
"Max":0,
"Used":0,
"Avail":512
},
{
"Name":"MEMP_UDP_PCB",
"Max":1,
"Used":0,
"Avail":512
}
],
"Connections":[]
},
"Code":0
}
I feel like I am having verification issues with the controller, maybe something with having to re verify every new ssh session into the instance with the controller is breaking something.
Sorry for the info dump, but this sums up what is happening. I am inclined to make a fresh EC2 instance, enroll it with the controller, and configure the same service as now. I am almost certain that everything will work fine, its when I disconnect and create a new ssh session is when the verification process is not clear between the controller and identity.
Thank you