Ziti-tunnel in linux not reconnecting on network disruption

Tunneler Apps Ziti Tunnel
1

I have ziti-tunnel running inside a docker application container .Laptop configured as Endpoint has 2 interfaces wired and wireless. If I disconnect the wired connection, network manager running in the laptop switches to wifi interface. We observe that ziti-tunnel not able to reconnect through wifi .
Is this expected? or please guide me how can I reconnect ziti successfully on network disruption.
Check below log for reference

[ 108.563] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:53638
[ 109.882] INFO sdk-golang/ziti.(*contextImpl).connectEdgeRouter: connection to edge router using token 648f378d-8e9b-4d35-b0d8-d09c061bbb3a
[ 110.629] INFO edge/tunnel.Run: {src-local=[100.64.0.4:5672] dst-local=[:1] dst-remote=[] src-remote=[100.64.0.4:53638]} tunnel started
[ 540.580] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 559.233] ERROR sdk-golang/ziti.(*contextImpl).runSessionRefresh: {error=[failed contact controller: Get “https://126a363c57-0e02-40d8-a25f-9704808c6ccb.production.netfoundry.io:443/current-api-session/service-updates”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)]} failed to check if service list update is available
[ 580.582] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 620.583] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 654.242] INFO edge/tunnel.myCopy: {src-local=[100.64.0.4:5672] dst-local=[:1] dst-remote=[] src-remote=[100.64.0.4:53638]} stopping pipe
[ 655.165] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:53970
[ 660.584] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 661.088] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[2]} timeout waiting for response
[ 665.178] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:53980
[ 667.098] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[3]} timeout waiting for response
[ 667.327] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 670.410] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[4]} timeout waiting for response
[ 675.185] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:53986
[ 675.873] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[5]} timeout waiting for response
[ 681.833] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 685.193] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54004
[ 686.834] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[6]} timeout waiting for response
[ 690.426] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[7]} timeout waiting for response
[ 692.293] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[8]} timeout waiting for response
[ 692.523] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 695.202] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54010
[ 695.887] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[9]} timeout waiting for response
[ 696.118] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 700.435] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[10]} timeout waiting for response
[ 700.585] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 705.212] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54016
[ 705.895] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[11]} timeout waiting for response
[ 706.131] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 710.443] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[12]} timeout waiting for response
[ 715.220] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54022
[ 715.902] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[13]} timeout waiting for response
[ 716.133] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 720.451] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[14]} timeout waiting for response
[ 725.230] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54028
[ 725.915] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[15]} timeout waiting for response
[ 731.031] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 735.238] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54054
[ 736.031] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[16]} timeout waiting for response
[ 740.465] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[17]} timeout waiting for response
[ 740.587] ERROR foundation/metrics.ProbeLatency [ch{ziti-sdk}->u{classic}->i{RrKyz}]: latency timeout after [10s]
[ 741.483] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[18]} timeout waiting for response
[ 741.709] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 745.247] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54060
[ 745.918] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[19]} timeout waiting for response
[ 746.145] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 750.474] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[20]} timeout waiting for response
[ 755.256] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54066
[ 755.926] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[21]} timeout waiting for response
[ 756.152] ERROR edge/tunnel.DialAndRun: {error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response] service=[RabbitAMQP]} tunnel failed
[ 760.512] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[22]} timeout waiting for response
[ 765.267] INFO edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept: received connection: 100.64.0.4:5672 → 100.64.0.4:54072
[ 765.968] ERROR sdk-golang/ziti/edge/impl.(*edgeConn).Connect: {connId=[23]} timeout waiting for response
[ 766.194] ERROR edge/tunnel.DialAndRun: {service=[RabbitAMQP] error=[unable to dial service ‘RabbitAMQP’: timeout waiting for response]
Thanks in Advance

2

Hi surekha1723,

After you switch over from wired to wireless, does that docker container still have internet access? Can you exec into it when these errors happen and try to curl something like discorse or github or anything?

Is this endpoint hosting rabbit or is it trying to contact rabbit? What version is running in the container, do you happen to know?

I can try to reproduce this myself but if you have any additional information about the overall setup it might help

3

Hi @TheLumberjack

Thanks for the reply. Please find my response inline

After you switch over from wired to wireless, does that docker container still have internet access?
------------> Yes, I have access to internet inside the container. I am able to do wget ,curl and install packages via internet as well

Is this endpoint hosting rabbit or is it trying to contact rabbit?
-----------> Endpoint is trying to access rabbitmq

What version is running in the container, do you happen to know?
------------>./ziti-tunnel version – v0.19.7(Assuming you asked about ziti)

Docker version: 20.10.7

4

Oh, sorry @surekha1723, I forgot to ask how the endpoint had the ziti-tunnel provisioned. Could you share how you provisioned the docker container? For example did you do it manually using your own docker file (if so I’d like the file) or did you use a guide such as this one which uses the docker image we push out to docker hub? I expect you used that guide but I’d just like to make sure. Assuming you used the compose file, did you make any changes to that compose file?

Also I was wondering if at this time of the change can you: curl -k https://126a363c57-0e02-40d8-a25f-9704808c6ccb.production.netfoundry.io:443/version? I just tried now and my DNS resolution of that url failed. Are you able to connect now? If so can you provide me the IP that resolves for you for that url? That is the correct URL, right?

5

Hi @TheLumberjack

Please find below url
https://6a363c57-0e02-40d8-a25f-9704808c6ccb.production.netfoundry.io:443/current-api-session/service-updates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]} failed to check if service list update is available.
It might have got changed while pasting sorry for that
I am not using ziti docker image. We dont want to provide --network=host . Please find below Dockerfile
FROM ubuntu
COPY ./ziti /ziti
apt-get -y install iptables

Here i have downloaded ziti-linux ,copying and building the image
Using that image i am running my container adding --cap-add=NET-ADMIN for iptables
After that i am running ziti-tunnel inside the container in which i am succesfull .I am able to access rabbitmq from the application running inside the same container.

6

Great. Thanks for the docker file / details. Gimme a bit and I’ll talk to some people and try to reproduce etc.

7

Thanks @TheLumberjack
Let me know if any other further details required

8

Hi @surekha1723, Looks like I was able to reproduce something that seems to be the same thing as you have reported. We’ll keep trying to narrow it down and understand it but it might take some time. I’ll follow up here when there’s more to share

9

@surekha1723 we think we have a fix for this particular issue and it should be in the next release. Releases happen “regularly” but I don’t know when it will be. The issue I filed is here ziti-tunnel with multiple adapters cannot always failover · Issue #420 · openziti/ziti · GitHub if you are interested.

I’ll try to remember to follow up here when that release hits but you can also track Releases · openziti/ziti · GitHub and watch for the next one.

Thanks for the bug report!