Testing SSH intercept: failed to dial fabric - invalid link destination

Support
1

Hi All,

Im using tproxy intercept on a local Ziti router and trying to forward traffic to a remote router on a VPC (contabo-router) which acts as the egress/terminator, this then has a nat rule where the traffic carries on to a to a raspberry Pi (10.12.10.9) where we connect over SSH (port 22).

This is part of a setup to transparently redirect and route SSH traffic from a docker container into a remote network using OpenZiti and the sidecar trick.

The actual ziti tunnels are running over an existing overlay / vpn where I'm using Zerotier, and I've verified end-to-end routing, DNS, and local iptables as well as even checking for MTU issues.

OpenZiti is working to the extent that intercepted traffic triggers attempts to build the connection and traffic is seen on the local end (via tcpdump) where the source ip is 10.12.10.9 and the dest ip is 10.12.10.9 and this goes into the loopback interface (assume this is the ingress part of sending it to the ziti tunnel)
However I thought it strange it has the source address of 10.12.10.9 which is the destination which kind of seems wrong? How can a source and a destination be the same? Yet traffic does actually get to the remote tunnel router (contabo) it does egress and the manual nat rule works and translates the IP and it carries on to the Pi which receives the traffic and responds, but then there is a timeout its like the client side never responds and something cuts it off mid conversation?

Also I see the following other items of interest:

When I run the command to show circuits, I never see any circuits created? is this normal behaviour? Even when running "watch" command to see if its a transient circuit thats created and then torn down I never see anything?

Terminators are up and running by the looks of things.

I have some errors seen in the logs along the lines of
"failed to dial fabric - invalid link destination"

My other issue is tha when I try with my snmp / udp /161 intercept, I see the traffic ingress to the loopback but this time nothing leaves the remote side, it like it just is lost in the ether and never appears at the other end. I'm guessing this is something different.

[root]# ziti fabric list links

╭────────────────────────┬────────────────────┬────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                     │ DIALER             │ ACCEPTOR           │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├────────────────────────┼────────────────────┼────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 4JNfJj13dgYpiJNLB45g5E │ local-router-lsk15 │ contabo-router     │           1 │       2.4ms │   65000.0ms │ Connected │     up │     65003 │
│ 7VDFWpQK2BNfEJSqNDlyyL │ contabo-router     │ local-router-lsk15 │           1 │       4.1ms │   65000.0ms │ Connected │     up │     65005 │
╰────────────────────────┴────────────────────┴────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-2 of 2
[wiz-contabo-srv-01:openziti-controller]
[root]# ziti edge list edge-routers

╭───────────┬────────────────────┬────────┬───────────────┬──────┬────────────╮
│ ID        │ NAME               │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├───────────┼────────────────────┼────────┼───────────────┼──────┼────────────┤
│ xUEQ-4UCo │ local-router-lsk15 │ true   │ true          │    0 │ public     │
│ xcPcN-mto │ contabo-router     │ true   │ true          │    0 │ public     │
╰───────────┴────────────────────┴────────┴───────────────┴──────┴────────────╯
results: 1-2 of 2
2

I know the issue its a network problem on my end I think. Will test and come back with the results.

3

Update, don't think it is my network now, as I just tried going to an IP out on the internet so it doesn't try and traverse my internal VPN, and I see the traffic is intercepted successfully and emerge from the server (terminator) at the remote end and go on to leave that server and carry on to another remote internet based ssh server and then the connection is killed in the same way where I get the immediate disconnect situation.

4 
025-05-23T17:19:46.473Z","timelineId":"wU2Oa3aHg"}
ziti-controller-1  | {"currentIndex":32,"file":"github.com/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github.com/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":32,"routerId":"xUEQ-4UCo","routerName":"local-router-lsk15","subscriptionDuration":"2025-05-23 17:24:46.474003325 +0000 UTC m=+15089.978083092","time":"2025-05-23T17:19:46.474Z","timelineId":"wU2Oa3aHg"}
ziti-controller-1  | {"_channels":["establishPath"],"attemptNumber":1,"circuitId":"QJvTVnltk","file":"github.com/openziti/ziti/controller/network/routesender.go:197","func":"github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/xcPcN-mto] for attempt [#0] of [s/QJvTVnltk] (invalid link destination 4JNfJj13dgYpiJNLB45g5E)","serviceId":"4VNjVPeiJ6Ew6AN9IuJYVy","time":"2025-05-23T17:21:41.702Z"}
ziti-controller-1  | {"_channels":["selectPath"],"attemptNumber":1,"circuitId":"QJvTVnltk","error":"error creating route for [s/QJvTVnltk] on [r/xcPcN-mto] (invalid link destination 4JNfJj13dgYpiJNLB45g5E)","file":"github.com/openziti/ziti/controller/network/network.go:650","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"4VNjVPeiJ6Ew6AN9IuJYVy","serviceName":"ssh-service","time":"2025-05-23T17:21:41.702Z"}
ziti-controller-1  | {"file":"github.com/openziti/ziti/controller/network/network.go:544","func":"github.com/openziti/ziti/controller/network.(*Network).LinkFaulted","level":"info","linkId":"4JNfJj13dgYpiJNLB45g5E","msg":"removing failed link","time":"2025-05-23T17:21:41.703Z"}
ziti-controller-1  | {"_context":"ch{xcPcN-mto}-\u003eu{classic}-\u003ei{xcPcN-mto/arza}","fault.iteration":0,"file":"github.com/openziti/ziti/controller/handler_ctrl/fault.go:144","func":"github.com/openziti/ziti/controller/handler_ctrl.(*faultHandler).handleFaultedLink","level":"info","link.iteration":24,"linkId":"4JNfJj13dgYpiJNLB45g5E","msg":"link fault","routerId":"xcPcN-mto","time":"2025-05-23T17:21:41.703Z"}
ziti-controller-1  | {"file":"github.com/openziti/ziti/controller/network/network.go:1003","func":"github.com/openziti/ziti/controller/network.(*Network).handleRerouteLink","level":"info","linkId":"4JNfJj13dgYpiJNLB45g5E","msg":"changed link","time":"2025-05-23T17:21:41.703Z"}
ziti-controller-1  | {"_context":"ch{xcPcN-mto}-\u003eu{classic}-\u003ei{xcPcN-mto/arza}","fault.iteration":0,"file":"github.com/openziti/ziti/controller/handler_ctrl/fault.go:149","func":"github.com/openziti/ziti/controller/handler_ctrl.(*faultHandler).handleFaultedLink","level":"info","linkId":"4JNfJj13dgYpiJNLB45g5E","msg":"link fault for unknown link","routerId":"xcPcN-mto","time":"2025-05-23T17:21:41.742Z"}
ziti-controller-1  | {"_channels":["establishPath"],"attemptNumber":2,"circuitId":"QJvTVnltk","file":"github.com/openziti/ziti/controller/network/routesender.go:197","func":"github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/xcPcN-mto] for attempt [#1] of [s/QJvTVnltk] (invalid link destination 4JNfJj13dgYpiJNLB45g5E)","serviceId":"4VNjVPeiJ6Ew6AN9IuJYVy","time":"2025-05-23T17:21:41.742Z"}
ziti-controller-1  | {"_channels":["selectPath"],"attemptNumber":2,"circuitId":"QJvTVnltk","error":"error creating route for [s/QJvTVnltk] on [r/xcPcN-mto] (invalid link destination 4JNfJj13dgYpiJNLB45g5E)","file":"github.com/openziti/ziti/controller/network/network.go:650","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"4VNjVPeiJ6Ew6AN9IuJYVy","serviceName":"ssh-service","time":"2025-05-23T17:21:41.742Z"}
ziti-controller-1  | {"_channels":["selectPath"],"attemptNumber":3,"circuitId":"QJvTVnltk","file":"github.com/openziti/ziti/controller/network/network.go:659","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"circuit creation failed after [2] attempts, sending cleanup unroutes","serviceId":"4VNjVPeiJ6Ew6AN9IuJYVy","serviceName":"ssh-service","time":"2025-05-23T17:21:41.742Z"}
ziti-controller-1  | {"_context":"ch{xUEQ-4UCo}-\u003eu{classic}-\u003ei{xUEQ-4UCo/ywVD}","error":"exceeded maximum [2] retries creating circuit [c/QJvTVnltk] (error creating route for [s/QJvTVnltk] on [r/xcPcN-mto] (invalid link destination 4JNfJj13dgYpiJNLB45g5E))","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:78","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"tunnel.create.circuit.v2","routerId":"xUEQ-4UCo","time":"2025-05-23T17:21:41.742Z"}
ziti-controller-1  | {"destRouterId":"xcPcN-mto","file":"github.com/openziti/ziti/controller/network/network.go:517","func":"github.com/openziti/ziti/controller/network.(*Network).NotifyExistingLink","iteration":25,"level":"info","linkId":"4JNfJj13dgYpiJNLB45g5E","msg":"router reported link added","routerId":"xUEQ-4UCo","time":"2025-05-23T17:21:41.792Z"}
5

I think its the advertised address for the ziti routers, let me try adjusting that...

6

OK fixed it, it seems all along it was because I hadnt set the advertise address correctly for the routers.

They were advertising ziti-router the default.

:man_facepalming: :man_facepalming: :man_facepalming:

Quick update to say it works for snmp aswell now :trophy: :trophy:

1 Like
7

Congrats on getting it working!

I just wanted to chime in with an explanation for the matching source and destination IPs on the sockets at the intercepting router. This is a side-effect of using TPROXY rules to intercept connections from local processes (vs inbound from other hosts).

The TPROXY rule can only be used for connections that are in the PREROUTING chain of the iptables mangle table. Connections from local processes won't traverse this chain unless a local route exists for the destination address. This is explained in a little more detail here.

1 Like