Issues with enrolling on windows desktop edge && 3rd Party CA issues

[k1nty]

Hello,

I recently made a post on the subreddit asking about the feasibility of my implementation. I've made some progress in getting a network set up on an EC2 instance and enrolling my windows host machine through the desktop edge and a separate Ubuntu EC2 instance on the network through a tunneler, and I was able to simulate communication between the two.

I began running into issues when following the Using 3rd Party CAs with Ziti CLI, and ZAC. I would create a new CA just fine, but when I tried to verify it through ZAC, the contents would not populate.

A blank shaded window would popup, but there would be no verification token field or standard UI layout. I would need to refresh the page to close out of it

After troubleshooting this for a while, it led me to doing a fresh ziti install on a new EC2 instance following the same steps as before. The network setup went as expected and accessing the ZAC console did as well. I created my identities through the CLI and exported JWT to my local machine using scp to enroll my windows device and separate EC2 instance in the network again. I was able to enroll the separate EC2 instance just fine using the ziti-edge-tunnel, but when I tried to enroll my windows device through the desktop edge I am getting a similar enrollment failed: CONTROLLER_UNAVAILABLE(-7) that has been seen on here before. Here is the full log.

[2024-11-08T16:04:15.504Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:2086 run() ============================================================================
[2024-11-08T16:04:15.511Z]   ERROR ziti-edge-tunnel:tun.c:107 WintunLogger() Failed to find matching adapter name: Element not found. (Code 0x00000490)
[2024-11-08T16:04:31.901Z]   ERROR ziti-sdk:ziti_enroll.c:249 well_known_certs_cb() D:/a/ziti-tunnel-sdk-c/ziti-tunnel-sdk-c/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:144 - ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)
[2024-11-08T16:04:31.901Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:330 tunnel_enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-7)

I began looking into the WintunLogger() Failed to find matching adapter through my adapter settings and this is what I found.

I've uninstalled and reinstalled the desktop edge client multiple times, but I am still facing the same error. I have yet to try this on another device, but plan to soon.

I've tried the steps from a previous post:

This can happen for a few different reasons. The most common of which are:

    the jwt issuer url is invalid
    the certificate presented by the iss url was not signed by the same key that signed the jwt and server cert
    the controller is offline or unreacable
    the JWT has expired. it's only valid for 3 hours by default. see the edge.enrollment.edgeIdentity.duration setting of the controller

My EXTERNAL_DNS configuration is through the public IP of my EC2 instance.

• I am unable to curl https://44.204.250.56:8441 from my windows machine or my already enrolled EC2 instance
• I am able to use openssl to return a certificate valid for the URL I am connecting to
• I can verify that my JWT has not expired

Following this discussion post Host OpenZiti Anywhere Unable to enroll - #11 by TheLumberjack, I was able to verify that the SSL handshake changes a bit between the IP and the DNS.

openssl s_client -connect ec2-44-204-250-56.compute-1.amazonaws.com:8441 </dev/null
CONNECTED(00000003)
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-root-ca Root CA
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-root-ca Root CA
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-intermediate
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111 server certificate
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111 server certificate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-intermediate
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov  8 13:48:04 2024 GMT; NotAfter: Nov  8 13:49:00 2025 GMT
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-root-ca Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov  8 13:47:51 2024 GMT; NotAfter: Nov  6 13:48:49 2034 GMT
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-root-ca Root CA
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ip-172-31-8-111-edge-controller-root-ca Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov  8 13:47:43 2024 GMT; NotAfter: Nov  6 13:48:42 2034 GMT
---
Server certificate

One other thing I noticed is that I currently have no sessions running in ZAC, there are API sessions for my Default Admin, Router, and my windows laptop accessing ZAC as the IPs match.

The reason I bring this up is I noticed a session was there when I previously had everything connected and running on the network.

I am sorry for the long post, but I wanted to make sure I included everything I've attempted up until this point. I really appreciate any help. Thank You!

[TheLumberjack]

Hi @k1nty, welcome to the community and hello from Reddit! You seem to be attacking much of OpenZiti quickly, nice!

Your post is long and has a few things for me to address, I'll try to catch them all...

@k1nty

but when I tried to verify it through ZAC, the contents would not populate.

I can confirm this is a ZAC bug. I had the same problem. (@rgalletto fyi, this breaks ZAC's 3rd party ca verification). The "easy" workaround is to use the ziti CLI. The "TOKEN" field is what you want/need to copy to move forward:

ziti edge list cas
╭────────────────────────┬───────────────┬───────┬────────────┬─────────────┬───────────────────────────────────────────────────────────────╮
│ ID                     │ NAME          │ FLAGS │ TOKEN      │ FINGERPRINT │ CONFIGURATION                                                 │
├────────────────────────┼───────────────┼───────┼────────────┼─────────────┼───────────────┬──────────────────────┬────────────────────────┤
│ 6yoA2Xup2pQNv8KsPmCAmv │ new_ca_182739 │ [OE]  │ d-iYxevxd  │ -           │ AutoCA        │ Identity Name Format │ [caName]-[commonName]  │
│                        │               │       │            │             │               ├──────────────────────┼────────────────────────┤
│                        │               │       │            │             │               │ Identity Roles       │                        │
╰────────────────────────┴───────────────┴───────┴────────────┴─────────────┴───────────────┴──────────────────────┴────────────────────────╯
results: 1-1 of 1

Flags: (V) Verified, (A) AutoCa Enrollment, (O) OttCA Enrollment, (E) Authentication Enabled

The instructions on the video are old too using older variables. You can 'fix' the missing variable by setting one var:

export ZITI_EDGE_CTRL_ADVERTISED="${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT}"

After that, you should be able to run the curl commands shown in description of the video and you can avoid using ZAC for this bit. You also need to be on the lookout for > commands if you copy from the notes of the video. They need to be converted to > as in this command where I had to change > (i think there are a few of these to be on the lookout for):

curl -sk https://${ZITI_EDGE_CTRL_ADVERTISED}/.well-known/est/cacerts > ${ZITI_PKI}/fetched-ca-certs.p7

Do note that at this time, the ziti desktop edge for windows (or Mac) does not 'natively' support enrolling 3rd party CAs identities, only the ziti cli or ziti-edge-tunnel should support them there just isn't any UI support yet. You can use them but you can't enroll them yet. You will have to add the identity manually after enrolling to C:\Windows\System32\config\systemprofile\AppData\Roaming\NetFoundry and then restart the ZDEW using the big green button on the ui (or net stop ziti & net start ziti)

You should never need to uninstall/reinstall the software either fwiw If you find yourself doing that, please just throw up a discourse post and we'll get you an answer asap...

I think I'll stop here and let you reply. Hopefully that gets you moving along again. I'll be offline all weekend but I may check in from time to time. If someone else sees your response and can help, I'm sure they will.

Cheers

[rgalletto]

I was able to verify the issue with displaying the "Verify Certificate" modal on the Certificate Authorities page. I have opened an issue in the ziti-console project here, and I have a fix in place pending review.

This should go out in the next release of ZAC sometime early next week. Thanks for finding and reporting!

[k1nty]

Thank for the response. I was able to modify the commands from the video per your instructions and create, verify, and bind a 3rd Party CA identity to a service that allowed me to curl to a server, similar to the video.

When running the ziti-edge-tunnel, I did this on the same instance my controller and edge router are configured on as that is where the CA cert was created and stored.

sudo ~/ziti-edge-tunnel run -i "$ZITI_PKI/$ca_name/keys/${identity_name}.json"

I was able to see the incoming curl request from my local machine to "server", which in the video there is a eth0.discourse.ziti, but I didn't have anything configured so my curl request returned

~$ curl Me.ziti
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

In future implementations, if I want to host services where two edge nodes separate from the instance that is hosting my controller and edge router can send UDP packets to only each other over a specific port, would I...

1. Create the identities for the two edge nodes and properly enroll them with the JWT
2. Create the 3rd Party CAs that are to be bound to each service on the instance hosting my controller and edge router
3. Create the proper host and intercept configs and link those services to the two identity attributes
4. What I am confused about is where do I run the ziti-edge-tunnel? Can I only run it on my controller instance as that is where the certs are stored? Will that be enough for the two edge nodes to communicate given proper host and intercept config, or do I need to transfer the certs to my other edge nodes and run the ziti-edge-tunnel there as well?

Sorry for the long post, I am still trying to wrap my head around where the ziti-edge-tunnel should be ran. Thank you

[TheLumberjack]

@k1nty

Sorry for the long post, I am still trying to wrap my head around where the ziti-edge-tunnel should be ran

I'll tackle this question first. In general, my/our guidance is to run the "tunneler" (ziti-edge-tunnel, or Ziti Desktop Edge for Mac/Windows, a tunneler-enabled router) "as close to the target service as possible". Often, that means you run the tunneler on the same machine with the service you're trying to secure/provide reach to. We have been calling this "Zero Trust Host Access (ZTHA)". This is conceptually true for the "client-side" as well as the "peer/server-side". So if you can't embed OpenZiti into the software on either side, you run the tunneler as close to that software as you can. This reduces your trust zone to the OS itself (on both sides). If you can't install software on the OS itself, then you have to fall back to "Zero Trust Network Access (ZTNA)" where you trust the entire private network, which is the "most trusting" of the zero trust models. Ideally, you'd want the zero trust principles baked into the application itself using one of our SDKs which reduces the zone of trust even more...

With that said - I can answer the first question you asked:

@k1nty

if I want to host services where two edge nodes separate from the instance that is hosting my controller and edge router can send UDP packets to only each other over a specific port,

You do not need a tunneler on the controller at all in this situation. You would install a tunneler on the first node as the "client" side and then you'd install a tunneler on the "peer/server" side. This would allow you to send traffic to the UDP host you declare in the intercept config and have it offload at the far tunneler onto the loopback interface (or wherever you need it to go). You would not need to have a tunneler on the controller nor the edge router if you don't want to.

Does that help? I hope it does. I'm also glad you were able to get the video to work with the minor corrections. Sounds like I should update it!

[k1nty]

Thank you again for the detailed response, your explanation cleared a lot things up for me and my understanding of OpenZiti is improving everyday! I was able to implement my previous question of:

if I want to host services where two edge nodes separate from the instance that is hosting my controller and edge router can send UDP packets to only each other over a specific port

but I am still running into some minor issues.

Right now I have three edge devices separate from my controller that each have tunnelers installed, one is my laptop with WDE enabled and the other two are an EC2 instance and a Raspberry Pi with the debian tunneler configuration. Following your explanation, I've created two similar services. One for communication from my laptop to the EC2 instance, and one for communication from my laptop to the RPi. In both services, the host.v1 consists of:

• Protocol: UDP
• Address: public IP of EC2 instance or RPi
• Port: 14550

While the intercept.v1 consists of:

• Port Ranges: 14550
• Address: 'EC2.ziti' or 'RPi.ziti'
• Protocol: UDP

Both services policies are bound to the respective edge nodes of either EC2 instance or RPi. While the dial for both is set to my laptops identity attribute, with the intention of my laptop accessing the service. For Example:

EC2.bind

• Service Attribute: EC2
• Identity Attribute: EdgeEC2

EC2.dial

• Service Attribute: EC2
• Identity Attribute: Me.laptop

Both of these services work and I can simulate UDP packets being transmitted from my laptop to either the EC2 instance or the RPi. I am doing this through netcat, but it only works sometimes...

Here is what it looks like when I first run the tunneler on the EC2 instance without trying to simulate data

EC2 Instance:~$ sudo ziti-edge-tunnel run -i /opt/openziti/etc/identities/EdgeEC2.json
About to run tunnel service... ziti-edge-tunnel
(2988)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2988)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.1.5 @g2120296(HEAD) starting at (2024-11-19T22:36:06.260)
RTNETLINK answers: File exists
(2988)[        0.000]   ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{ip route add 100.64.0.0/10 dev ziti1} failed: 512/0/Success

(2988)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.6)
(2988)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2988)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:887 make_socket_path() effective group set to 'ziti' (gid=988)
(2988)[        0.014]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/EdgeEC2.json] is not loaded yet or already removed.
(2988)[        0.014]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2988)[        0.014]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1121 load_ziti_async() attempting to load ziti instance[/opt/openziti/etc/identities/EdgeEC2.json]
(2988)[        0.014]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1128 load_ziti_async() loading ziti instance[/opt/openziti/etc/identities/EdgeEC2.json]
(2988)[        0.014]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:402 load_id_cb() identity[/opt/openziti/etc/identities/EdgeEC2.json] loaded
(2988)[        0.017]    INFO ziti-sdk:ziti.c:437 ziti_start_internal() ztx[0] using tlsuv[v0.32.6/OpenSSL 3.3.1 4 Jun 2024]
(2988)[        0.017]    INFO ziti-sdk:ziti_ctrl.c:593 ziti_ctrl_init() ctrl[(null):] using https://40.201.235.56:8441
(2988)[        0.017]    INFO ziti-sdk:ziti.c:507 ztx_init_controller() ztx[0] Loading ziti context with controller[https://40.201.235.56:8441]
(2988)[        0.058]    INFO ziti-sdk:ziti.c:1759 version_pre_auth_cb() ztx[0] connected to Legacy controller https://40.201.235.56:8441 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2988)[        0.073]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:968 on_ziti_event() ziti_ctx[EdgeEC2] connected to controller
(2988)[        0.073]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:440 on_event() ztx[/opt/openziti/etc/identities/EdgeEC2.json] context event : status is OK
(2988)[        0.100]    INFO ziti-sdk:channel.c:270 new_ziti_channel() ch[0] (ip-172-31-8-111-edge-router) new channel for ztx[0] identity[EdgeEC2]
(2988)[        0.100]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1039 on_ziti_event() ztx[EdgeEC2] added edge router ip-172-31-8-111-edge-router@40.201.235.566
(2988)[        0.100]    INFO ziti-sdk:channel.c:799 reconnect_channel() ch[0] reconnecting NOW
(2988)[        0.119]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:940 on_service() hosting server_address[udp:3.90.72.176:14550] service[EC2]
(2988)[        0.119]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:563 on_event() =============== service event (added) - EC2:67d3ZGVJpEQZQYRuDC6VaG ===============
(2988)[        0.119]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
RTNETLINK answers: File exists
Command failed /tmp/ziti-tunnel-routes.oAeLGm:1
(2988)[        0.121]   ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{ip -force -batch /tmp/ziti-tunnel-routes.oAeLGm} failed: 256/0/Success

(2988)[        0.121]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2988)[        0.135]    INFO ziti-sdk:channel.c:697 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2988)[        0.135]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1043 on_ziti_event() ztx[DroneEC2] router ip-172-31-8-111-edge-router connected
(2988)[        0.528]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(2988)[        1.069]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

Then in a separate ssh session for the EC2 instance, as well as a terminal session on my laptop, I am able to simulate connections like so. Where I treat the instance as a server listening for on port 14550 for the intercept address EC2.ziti.

Laptop:~$ nc -u EC2.ziti 14550
Hello
How are you?
That is good
nice
cool beans
:)
:(
wooo
wo
wo
^C

EC2 Instance:~$ nc -u -l 14550
Hello
How are you?
That is good
nice
cool beans
:)
:(
^C

Here is what I will see in my tunneler:

(2988)[       39.471]    INFO tunnel-cbs:ziti_hosting.c:637 on_hosted_client_connect() hosted_service[Drone] client[Me] client_src_addr[udp:100.64.0.1:59326] dst_addr[udp:3.90.72.176:14550]: incoming connection
(2988)[       44.497]    INFO tunnel-cbs:ziti_hosting.c:637 on_hosted_client_connect() hosted_service[Drone] client[Me] client_src_addr[udp:100.64.0.1:59326] dst_addr[udp:3.90.72.176:14550]: incoming connection

But most of the time, the tunneler does not register every new message as an incoming connection, and will eventually throw the error below.

(2988)[       173.736]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input () br[0.5] err = -111/connection refused

Sometimes if I try to send another message a couple minutes later it will go through.

Could this be something network related on my end? I was thinking it could be due to the Listen Options I have in my host.v1 config, but that is empty. I only showed the output from the EC2 instance, but this also happens when communicating with the RPi in similar fashion.

Sorry again for the long post, it's been fun and rewarding as I continue to learn OpenZiti. Thank You

[TheLumberjack]

Thanks for the details. I feel like it painted the right picture for me. Assuming I've understood, it sounds like you have things mostly working but you're running into this problem

@k1nty

Sometimes if I try to send another message a couple minutes later it will go through.

This makes me think there are numerous connections in close wait. Can you run the test again but keep track of the os open sockets using ss or netstat? There's also an IP dump subcommand for the ziti-edge-tunnel that might be useful.

@scareything that sound right to you? Anything else that might be useful to understand here to help out?

[k1nty]

I was also leaning towards numerous connections, as when managing the current sessions in ZAC, I can see multiple sessions and API sessions for the same identity/service. Could this be due to the multiple ssh connections I have to the instance, even though only one of the them is running the tunneler. The Drone.EC2 service has two sessions created right after each other.

I ran another test today and was able to confirm connections using netstat locally on the instance.

You can see that on the instance, there are initially two connections for port 14550, but then when I send the message 'splendid' from my local machine, it does not reach the instance. Running netstat again shows that one of the connections dropped for some reason.

I am still running into the same issue of my tunneler not registering incoming connections for messages sent, even tho I was successfully communicating with the EC2 instance from my local machine using netcat.

I ran a status on the tunneler after and this is what was returned. Looks like all the messages I sent are actually logging -111/connection refused

systemctl status ziti-edge-tunnel
● ziti-edge-tunnel.service - Ziti Edge Tunnel
     Loaded: loaded (/usr/lib/systemd/system/ziti-edge-tunnel.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-11-19 17:24:06 UTC; 22h ago
   Main PID: 599 (ziti-edge-tunne)
      Tasks: 6 (limit: 1130)
     Memory: 12.7M (peak: 13.4M)
        CPU: 23.441s
     CGroup: /system.slice/ziti-edge-tunnel.service
             └─599 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

Nov 19 21:01:00 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    13013.422]    WARN ziti-sdk:ziti.c:1801 ztx_auth_state_cb() auth error: The request could not be completed. The session is not authorized or the credentials are invalid
Nov 19 22:32:09 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18482.711]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.25] err = -111/connection refused
Nov 19 22:32:12 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18485.916]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.26] err = -111/connection refused
Nov 19 22:34:08 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18601.346]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.28] err = -111/connection refused
Nov 19 22:39:08 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18901.642]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.31] err = -111/connection refused
Nov 19 22:39:14 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18907.583]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.32] err = -111/connection refused
Nov 19 22:39:37 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    18930.799]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.33] err = -111/connection refused
Nov 20 15:21:28 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    79041.852]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.35] err = -111/connection refused
Nov 20 15:21:39 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    79052.641]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.36] err = -111/connection refused
Nov 20 15:22:07 ip-172-31-1-229 ziti-edge-tunnel[599]: (599)[    79081.162]    WARN ziti-sdk:conn_bridge.c:386 on_udp_input() br[0.37] err = -111/connection refused

Could the tunneler be having issues generating a valid handshake with the controller each time I ssh into the instance? Even though I can see valid API sessions as seen above. I tried another session and was able to talk from my laptop to the instance successfully. Here is what was returned for netstat and ip_dump. The tunneler is not still not logging incoming connections, but it did log my ip_dumps.

~$ netstat | grep 14550
udp        0      0 ip-172-31-1-229.e:14550 ec2-3-92-74-179.c:40772 ESTABLISHED
udp        0      0 ip-172-31-1-229.e:40772 ec2-3-92-74-179.c:14550 ESTABLISHED
ubuntu@ip-172-31-1-229:~$ sudo ziti-edge-tunnel ip_dump
{
  "Success":true,
  "Data":{
    "Pools":[
      {
        "Name":"MEMP_PBUF_POOL",
        "Max":1,
        "Used":0,
        "Avail":1024
      },
      {
        "Name":"MEMP_TCP_PCB",
        "Max":0,
        "Used":0,
        "Avail":512
      },
      {
        "Name":"MEMP_UDP_PCB",
        "Max":1,
        "Used":0,
        "Avail":512
      }
    ],
    "Connections":[]
  },
  "Code":0
}

I feel like I am having verification issues with the controller, maybe something with having to re verify every new ssh session into the instance with the controller is breaking something.

Sorry for the info dump, but this sums up what is happening. I am inclined to make a fresh EC2 instance, enroll it with the controller, and configure the same service as now. I am almost certain that everything will work fine, its when I disconnect and create a new ssh session is when the verification process is not clear between the controller and identity.

Thank you

[TheLumberjack]

Hi @k1nty - thanks for all these details. I must admit though, I'm getting overwhelmed with where we are at...

Few things...: can we boil this back to basics? What is the current issue that you're experiencing? I believe that you're having intermittent connectivity issues and can't figure out 'why'. That's my high-level expectation of the problem so far...

Next, can we maybe make a whole new thread? This is no longer relevant to enrolling on windows nor 3rd party CA.

Can you give me the steps to reproduce the problem locally? If not, I understand but I'm just having a difficult time with understanding the current state, what you did and what went wrong...

Could the tunneler be having issues generating a valid handshake with the controller each time I ssh into the instance?

I would be shocked if this was relevant.

The tunneler is not still not logging incoming connections

this makes me think the traffic isn't getting into the overlay properly -- hence why I'd liike to be able to reproduce...

Sorry, I'd like to help but I admit I've "lost the plot" here. I am hoping a new thread will help... Thanks

[k1nty]

Sounds good, I'll create a new thread and try to detail the steps as best as possible.

Thank you again for the help.