Android Tunneler Bug

Tunneler Apps Ziti Mobile Edge for Android
1

I've noticed a bug with the Android tunneler: If the app is closed entirely (you can force stop it in Android Settings under Apps) the moment you open it and turn the service back on and try to authenticate (I'm using an IDP) you cannot reach the IDP website - but you also cannot access any internet websites, almost as if DNS is broken. My ping utility shows "Unknown host" and browser displays a DNS error.

I've experience this other times but it's the most "reproducible" method. If you then disable, and re-enable the identity in the tunneler app, you will be able to succesfully use the internet and authenticate via IDP.

I'm using a Pixel 8 Pro Android phone.

2

You actually beat me to posting about this :slight_smile:

I have noticed with two services (matrix and IMAP) that the ziti edge tunnels (android and linux respectively), fail to resolve.
If I hard-restart the tunnel, everything resolves again. I can access services, but it seems like the tunnels have something like a stale handle.

1 Like
3

I feel like there are 2 separate issues here, and I've experienced both as well (I mentioned one in another thread).

  1. DNS/Resolving/Internet breaking upon first Android client login (nothing resolves - and I'm not yet authenticated so apps don't work at all, as expected, but the IDP and normal browsing break), that's this issue.

  2. The second is my other thread where I have a similar issue to you, where I get logged in and some apps do work, and others do not - but yet DNS resolves to OpenZiti correctly, it's just not allowing all apps to work.

Although I seem to find these bugs, I do just want to state Open Ziti is great and has so much potential! I've been growing attached as I use it and learn more about how it works.

4

@qrkourier Is this/are these known issues, or would it be good to open an issue (or two) on GitHub?

5

Does "About" show v0.13.5? That's the version I'm trying to reproduce with on a Pixel 6.

I ignored the "app may misbehave" warning and force stopped ZME in Android Settings, but I didn't encounter the DNS problem. Can you trigger this DNS problem without "force stop" (which may lead to a non-graceful exit and a broken app state)? You mentioned "first Android client login" and that could mean you're encountering this DNS problem when you first turn on the tunnel in the ZME app. If so, does the DNS problem persist, or is it only manifesting "at first login?"

Thanks for clarifying that the workaround for the DNS issue is to toggle the tunnel in the ZME app.

The other issue was described as "some apps do work, and others do not." Are you saying that some Ziti services work, and others do not? Specifically, of the apps that are working, are they trying unsuccessfully to connect to a Ziti service? What about those apps that are not working?

6

Yes, latest version of all software is being used.

Just tested and confirmed it happens if I reboot the phone as well, and try to bring up the tunnel. There could be other circumstances but none come to mind right now.

I have a Windows Edge Client thread here, where @TheLumberjack confirmed the issue during sleep (again, may be triggered in other circumstances but this was the easiest way to reproduce it). I will let @thedarkula confirm the Android/Linux side of this. But yes, you're correct in that some ziti services work, others do not - but the ones that work, have no issues - they are intercepted and connect to the app just fine, and the ones that do not work will resolve to their standard (non ziti-tunneled) destination.

1 Like
7

this makes little sense to me because DNS queries not covered by intercepts are proxied to the DNS server provided by your network. Also they would not depend on any identity being authenticated via IdP or otherwise.

if you encounter that condition, please submit application feedback bundle before force-quitting the app (that should never be necessary)

1 Like
8

Just submitted one after a reboot and re-occurrence :slight_smile:

9

Ok, so I tried clearing the cache in Android settings and clearing the data to start fresh, re-enrolled and right on the first connection, when it loads the IDP the internet is not working - I then disable/re-enable and i'm able to login again. It seems like it is some sort of bug maybe specific to the Pixel device and OpenZiti. Given the previous logs didn't have much in them - is there something else I should be checking?

10

It does. The commit shows c5e860f.

Something else of note is that, specifically on Linux, I noticed that ziti-edge-tunnel has a severe memory leak.

The percent memory usage dropped from 70.2% to 31.2% following a restart of ziti-edge-tunnel.

The systemd logs show this just before restarting the service:

Apr 07 07:56:09 computer-name ziti-edge-tunnel[933]: (933)[   207515.754]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3253]
Apr 07 07:56:19 computer-name ziti-edge-tunnel[933]: (933)[   207525.756]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:40962 err=-14, terminating connection
Apr 07 07:56:19 computer-name ziti-edge-tunnel[933]: (933)[   207525.756]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3254]
Apr 07 07:56:29 computer-name ziti-edge-tunnel[933]: (933)[   207535.757]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:51958 err=-14, terminating connection
Apr 07 07:56:29 computer-name ziti-edge-tunnel[933]: (933)[   207535.757]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3255]
Apr 07 07:56:39 computer-name ziti-edge-tunnel[933]: (933)[   207545.758]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:53044 err=-14, terminating connection
Apr 07 07:56:39 computer-name ziti-edge-tunnel[933]: (933)[   207545.758]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3256]
Apr 07 07:56:49 computer-name ziti-edge-tunnel[933]: (933)[   207555.759]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:35700 err=-14, terminating connection
Apr 07 07:56:49 computer-name ziti-edge-tunnel[933]: (933)[   207555.759]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3257]
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.201]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:36368 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.201]    WARN ziti-sdk:connect.c:222 close_conn_internal() conn[0.3179/lNMEp26N/Closed](jellyfin) dumping 24 bytes of undelivered data
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.202]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:57346 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.202]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3259]
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.253]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:40264 err=-14, terminating connection
Apr 07 07:57:07 computer-name ziti-edge-tunnel[933]: (933)[   207574.253]    WARN ziti-sdk:connect.c:222 close_conn_internal() conn[0.3221/UNJ3fWNo/Closed](jellyfin) dumping 13026546579 bytes of undelivered data
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]   ERROR tunnel-sdk:tunnel_tcp.c:190 on_tcp_client_err() client=tcp:100.64.0.1:43354 err=-14, terminating connection
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3118]
Apr 07 08:02:04 computer-name ziti-edge-tunnel[933]: (933)[   207871.216]    WARN ziti-sdk:channel.c:559 dispatch_message() ch[21] received message without conn_id or for unknown connection ct[ED72] conn_id[3118]

Here is a relevant screenshot from the output of glances:
Ziti Tunnel High Memory-pre-restart

11

@thedarkula - can you make a separate thread for the suspected memory leak please? thanks

12

Done :slight_smile:

13

Have the logs that @007bond007 supplied shed any light on the situation?

This is still occuring for me with the latest Android and Linux tunnels.

14

Also wondering this :slight_smile:

15

issue should be resolved with next release

16

The fix is how available via the testing channel

17

Hi @ekoby , on the Github page it says " DNS bypass not working without any identities" - is this the same bug, where if you DO have identities/services the DNS breaks the first time you connect to Ziti, but then subsequent attempts, DNS is working again?

18

Yes, it is the same.

1 Like
19

Thanks @ekoby this issue is now fixed! (Initial DNS not working), great work!. The 2nd issue in my other thread is still happening (Some intercepts not working, some are until restarting service/identity).

20

are you adding/editing services/intercepts and expecting them to work?

this could be just a delay in updating services to your endpoint/device. This is currently done with polling. On the phones we reduce polling rate to conserve the battery.

Bringing up ZME interface forces the refresh. Can you try you services after opening UI (without identity restart)?