Fixed it.
Had to change it to run as root, not sure whether thats a terrible thing!?
#user: ${ZIGGY_UID:-2171}
user: "0:0"
Two things:
- console
- tproxy
First, the console. 
Good news: you're already using the openziti/ziti-controller Docker image to run your Ziti controller, and it comes with the console, so you don't need a separate container for the console. That was a legacy standalone console that can be fully replaced by the new built-in console.
To use the built-in console, navigate to the URL of your controller's management API. The default is the same URL as the controller's advertised host and port with the path /zac/.
Second topic: tproxy router mode
You're troubleshooting some problems with the tproxy tunneling mode for the Ziti router you're running with openziti/ziti-router Docker image, right?
There are several strategies for different kinds of tunneling that differ by which direction you're tunneling and whether you want to tunnel just a container, just a process, or the entire OS. If you share a little about the goal for tunneling we might have time saving advice, but for now I'll assume you want your router to provide Ziti DNS and transparent proxy running as a sidecar for some other client application container that needs to connect to Ziti services. That's the main use case for tproxy router mode in Docker.
Take a look at the sidecar examples here because they detail the permissions as well as the run-as root requirement you discovered: Deploy the Router with Docker | OpenZiti
Here's a compose snip that shows only the tproxy bits of the router service/container that get merged in as a compose override in that example.
See how the client app uses the router as a sidecar to access Ziti services?
services:
ziti-router:
dns:
- 127.0.0.1
- 1.1.1.1
user: root
cap_add:
- NET_ADMIN
my-client-app:
image: some-app-image
network_mode: service:ziti-router
depends_on:
ziti-router:
condition: service_healthy
1 Like
Thanks I'll try that, yes I'm trying to re-route traffic based on port and protocal to a new destination, initially testing ssh but would like to also get snmp polling working.
This is all in containers
Originally I was using a cobbled together combination of wireguard for the "overlay", policy based routing and firewall marks in FRR (free range routing), and BGP with BFD for HA and failover between two of the overlay tunnels. I'm hoping I can simplfy all that by using Ziti which I tihnk could replace wireguard and PBR and possibly acheive some kind of HA aswell.
In theory I could have used something like GRE tunnels instead of wireguard for the overlay bit too but its not encrypted.
do I have a problem here:
[root]# ziti edge list edge-routers
╭───────────┬────────────────────┬────────┬───────────────┬──────┬──────────────────╮
│ ID │ NAME │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├───────────┼────────────────────┼────────┼───────────────┼──────┼──────────────────┤
│ Yk0KxGqfw │ contabo-router2 │ true │ true │ 0 │ contabo │
│ │ │ │ │ │ contabo-router │
│ lDOk2cqTY │ local-router-lsk15 │ true │ true │ 0 │ local-lsk-router │
╰───────────┴────────────────────┴────────┴───────────────┴──────┴──────────────────╯
results: 1-2 of 2
[wiz-contabo-srv-01:openziti-router]
[root]# ziti edge list service-policies
ziti edge list terminators
╭────────────────────────┬───────────────────┬──────────┬───────────────┬─────────────────────────────────────────┬─────────────────────╮
│ ID │ NAME │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES │ POSTURE CHECK ROLES │
├────────────────────────┼───────────────────┼──────────┼───────────────┼─────────────────────────────────────────┼─────────────────────┤
│ 2teooMpYV8YQatplexKrcH │ dialer-ssh-policy │ AnyOf │ @ssh-service │ #lsk @local-router-lsk15 │ │
│ 6wDH6RWLZsLi1t3yVa8Zwy │ bind-ssh-policy │ AnyOf │ @ssh-service │ #contabo #contabo-router @Default Admin │ │
╰────────────────────────┴───────────────────┴──────────┴───────────────┴─────────────────────────────────────────┴─────────────────────╯
results: 1-2 of 2
╭────┬─────────┬────────┬─────────┬─────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├────┼─────────┼────────┼─────────┼─────────┼──────────┼──────┼────────────┼──────────────┤
╰────┴─────────┴────────┴─────────┴─────────┴──────────┴──────┴────────────┴──────────────╯
results: none
[wiz-contabo-srv-01:openziti-router]
[root]# ziti edge list identities
╭───────────┬────────────────────┬─────────┬────────────────────────┬─────────────╮
│ ID │ NAME │ TYPE │ ATTRIBUTES │ AUTH-POLICY │
├───────────┼────────────────────┼─────────┼────────────────────────┼─────────────┤
│ I0sZe3rsl │ Default Admin │ Default │ contabo,contabo-router │ Default │
│ VZPfaJKfY │ jon │ Default │ │ Default │
│ lDOk2cqTY │ local-router-lsk15 │ Router │ lsk │ Default │
╰───────────┴────────────────────┴─────────┴────────────────────────┴─────────────╯
I have two routers but only one of my identities seems to show up as a router?
Also I don't see any terminators?
I can see the top mangle rule in iptables intercepting traffic which should forward it to the tproxy rule but the tproxy rule stays as 0 and doesn't increment?
f23c5d3bd6c:~# iptables -L -vvt mangle
ip mangle PREROUTING 3
[ counter pkts 2284 bytes 245218 ]
[ immediate reg 0 jump -> NF-INTERCEPT ]
ip mangle NF-INTERCEPT 8
[ payload load 1b @ network header + 9 => reg 1 ]
[ cmp eq reg 1 0x00000006 ]
[ match name comment rev 0 ]
[ payload load 2b @ transport header + 2 => reg 1 ]
[ cmp eq reg 1 0x00001600 ]
[ counter pkts 0 bytes 0 ]
[ target name TPROXY rev 1 ]
Chain PREROUTING (policy ACCEPT 2284 packets, 245K bytes)
pkts bytes target prot opt in out source destination
2284 245K NF-INTERCEPT all -- any any anywhere anywhere
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain NF-INTERCEPT (1 references)
pkts bytes target prot opt in out source destination
0 0 TPROXY tcp -- any any anywhere anywhere /* ssh-service */ tcp dpt:ssh TPROXY redirect 127.0.0.1:36107 mark 0x1/0x1
Oh and I did a dial type policy on the intercept / local side and a bind type policy on the remote / head-end (egress)
Is that the right way around?
Created another intercept service with a specific ip in there and that seems to intercept the traffic but fails with the dial part:
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:555","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).addInterceptAddr","level":"info","msg":"Adding rule iptables -t mangle -A NF-INTERCEPT [-m comment --comment ssh-firewall-service -d 10.12.10.5/32 -p tcp --dport 22:22 -j TPROXY --tproxy-mark 0x1/0x1 --on-ip=127.0.0.1 --on-port=38335]","time":"2025-05-21T13:37:01.197Z"}
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:310","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP","level":"info","msg":"received connection: 10.12.10.5:22 --\u003e 10.12.10.5:54794","time":"2025-05-21T14:02:54.024Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","error":"invalid edge router for session","file":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2/fabric.go:115","func":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2.(*fabricProvider).TunnelService","level":"warning","msg":"failed to dial fabric","service":"ssh-firewall-service","time":"2025-05-21T14:02:54.046Z"}
ziti-router-1 | {"error":"invalid edge router for session","file":"github.com/openziti/ziti/tunnel/tunnel.go:49","func":"github.com/openziti/ziti/tunnel.DialAndRun","level":"error","msg":"tunnel failed","service":"ssh-firewall-service","time":"2025-05-21T14:02:54.046Z"}
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:310","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP","level":"info","msg":"received connection: 10.12.10.5:22 --\u003e 10.12.10.5:54798","time":"2025-05-21T14:03:00.418Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","error":"invalid edge router for session","file":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2/fabric.go:115","func":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2.(*fabricProvider).TunnelService","level":"warning","msg":"failed to dial fabric","service":"ssh-firewall-service","time":"2025-05-21T14:03:00.438Z"}
ziti-router-1 | {"error":"invalid edge router for session","file":"github.com/openziti/ziti/tunnel/tunnel.go:49","func":"github.com/openziti/ziti/tunnel.DialAndRun","level":"error","msg":"tunnel failed","service":"ssh-firewall-service","time":"2025-05-21T14:03:00.438Z"}
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:310","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP","level":"info","msg":"received connection: 10.12.10.5:22 --\u003e 10.12.10.5:54808","time":"2025-05-21T14:03:01.108Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","error":"invalid edge router for session","file":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2/fabric.go:115","func":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2.(*fabricProvider).TunnelService","level":"warning","msg":"failed to dial fabric","service":"ssh-firewall-service","time":"2025-05-21T14:03:01.129Z"}
ziti-router-1 | {"error":"invalid edge router for session","file":"github.com/openziti/ziti/tunnel/tunnel.go:49","func":"github.com/openziti/ziti/tunnel.DialAndRun","level":"error","msg":"tunnel failed","service":"ssh-firewall-service","time":"2025-05-21T14:03:01.129Z"}
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:310","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP","level":"info","msg":"received connection: 10.12.10.5:22 --\u003e 10.12.10.5:54812","time":"2025-05-21T14:03:01.768Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","error":"invalid edge router for session","file":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2/fabric.go:115","func":"github.com/openziti/ziti/router/xgress_edge_tunnel_v2.(*fabricProvider).TunnelService","level":"warning","msg":"failed to dial fabric","service":"ssh-firewall-service","time":"2025-05-21T14:03:01.789Z"}
Can you have a look at your SERP (service-edge-router-policy)? It looks like the router isn't authorized to handle traffic for that service.
Run policy-advisor for that service name: ziti edge policy-advisor services ... and let's see what it reports?
Will try that now.
Also I'm not sure if I'm using something in the way its not supposed to.
If I put in a big network range like 10.0.0.0/8 which happens to stamp on my existing internal range that the routers are on it seems to go dead and start timing out...
ziti-router-1 | {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:555","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).addInterceptAddr","level":"info","msg":"Adding rule iptables -t mangle -A NF-INTERCEPT [-m comment --comment ssh-firewall-service -d 10.0.0.0/8 -p tcp --dport 22:22 -j TPROXY --tproxy-mark 0x1/0x1 --on-ip=127.0.0.1 --on-port=35703]","time":"2025-05-21T15:22:27.891Z"}
ziti-router-1 | {"error":null,"file":"github.com/openziti/ziti/tunnel/intercept/svcpoll.go:260","func":"github.com/openziti/ziti/tunnel/intercept.(*ServiceListener).addService","level":"warning","msg":"service is hostable but no compatible host config found. supported types: [host.v2, host.v1, ziti-tunneler-server.v1]","serviceId":"3Qz31WsFnuOOlSZCHFY1qY","serviceName":"ssh-firewall-service","time":"2025-05-21T15:22:27.893Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm}","error":"tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 172.21.0.2:33718-\u003e10.60.0.120:1280: write: broken pipe","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_impl.go:42","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"error closing peer after rx error","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm}","error":"read tcp 172.21.0.2:33718-\u003e10.60.0.120:1280: read: connection reset by peer","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_impl.go:44","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"rx error. closed peer and starting reconnection process","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm}","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_impl.go:171","func":"github.com/openziti/channel/v3.(*reconnectingImpl).pingInstance","level":"info","msg":"starting","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm}","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_impl.go:175","func":"github.com/openziti/channel/v3.(*reconnectingImpl).pingInstance","level":"info","msg":"exiting","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm} @tls:ziti-controller.wizznet.co.uk:1280","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_dialer.go:103","func":"github.com/openziti/channel/v3.(*reconnectingDialer).Reconnect","level":"error","msg":"unable to ping (use of closed network connection)","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"change":"Disconnected","ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","file":"github.com/openziti/ziti/router/state/manager.go:220","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).manageRouterDataModelSubscription","level":"info","msg":"currently subscribed controller has changed, resubscribing","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm}","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_impl.go:70","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Tx","level":"error","msg":"tx error (use of closed network connection). starting reconnection process","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"_context":"u{reconnecting}-\u003ei{NetFoundry Inc. Client jt9fMM0XZ/Bzwm} @tls:ziti-controller.wizznet.co.uk:1280","file":"github.com/openziti/channel/v3@v3.0.39/reconnecting_dialer.go:135","func":"github.com/openziti/channel/v3.(*reconnectingDialer).Reconnect","level":"error","msg":"reconnection attempt [#1] failed (dial tcp 10.60.0.120:1280: connect: connection refused)","time":"2025-05-21T15:22:38.532Z"}
ziti-router-1 | {"ctrlId":"NetFoundry Inc. Client jt9fMM0XZ","currentIndex":54,"error":"timeout waiting for message to be written to wire: context deadline exceeded","file":"github.com/openziti/ziti/router/state/manager.go:304","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"error","msg":"error to subscribing to router data model changes","renew":false,"time":"2025-05-21T15:22:43.533Z"}
[root]# ziti edge policy-advisor services
Policy General Guidelines
In order for an identity to dial or bind a service, the following must be true:
- The identity must have access to the service via a service policy of the correct type (dial or bind)
- The identity must have access to at least one on-line edge router via an edge router policy
- The service must have access to at least one on-line edge router via a service edge router policy
- There must be at least one on-line edge router that both the identity and service have access to.
Policy Advisor Output Guide:
STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR.
ID = identity name
ID ROUTERS = number of routers accessible to the identity via edge router policies.
- See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
SVC = service name
SVC ROUTERS = number of routers accessible to the service via service edge router policies.
- See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
DIAL_OK = indicates if the identity has permission to dial the service.
- See service polices for a service : ziti edge controller list service service-policies <service>
- See service polices for an identity: ziti edge controller list identity service-policies <identity>
BIND_OK = indicates if the identity has permission to bind the service.
ERROR_LIST = if the status is ERROR, error details will be listed on the following lines
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: Default Admin (1) -> ssh-service (0) Common Routers: (0/0) Dial: N Bind: Y
- Service has no edge routers assigned. Adjust service edge router policies.
ERROR: local-router-lsk15 (1) -> ssh-service (0) Common Routers: (0/0) Dial: Y Bind: N
- Service has no edge routers assigned. Adjust service edge router policies.
ERROR: Default Admin (1) -> ssh-firewall-service (0) Common Routers: (0/0) Dial: Y Bind: Y
- Service has no edge routers assigned. Adjust service edge router policies.
ERROR: jon (0) -> ssh-firewall-service (0) Common Routers: (0/0) Dial: Y Bind: Y
- Identity has no edge routers assigned. Adjust edge router policies.
- Service has no edge routers assigned. Adjust service edge router policies.
ERROR: local-router-lsk15 (1) -> ssh-firewall-service (0) Common Routers: (0/0) Dial: Y Bind: Y
- Service has no edge routers assigned. Adjust service edge router policies.
Do I need Servic/Router Policies? I have not set them yet? Only service policies and router policies are set.
Aha!
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: Default Admin (1) -> ssh-service (0) Common Routers: (0/0) Dial: N Bind: Y
- Service has no edge routers assigned. Adjust service edge router policies.
ERROR: local-router-lsk15 (1) -> ssh-service (0) Common Routers: (0/0) Dial: Y Bind: N
- Service has no edge routers assigned. Adjust service edge router policies.
OKAY : Default Admin (1) -> ssh-firewall-service (2) Common Routers: (1/1) Dial: Y Bind: Y
ERROR: jon (0) -> ssh-firewall-service (2) Common Routers: (0/0) Dial: Y Bind: Y
- Identity has no edge routers assigned. Adjust edge router policies.
OKAY : local-router-lsk15 (1) -> ssh-firewall-service (2) Common Routers: (1/1) Dial: Y Bind: Y
after adding a service/router policy!
update: just tried again and getting same error as before regarding invalid edge router for session
I actually don't know where that is from. I suspect you're running ZAC from a docker container (node-based) as opposed to delivering it from the controller? My "Policies" shows only three entries:
To answer the question, there are three policy types:
- service-policies (authorizes identities to services)
- edge-router-policies (authorizes identities to use edge routers)
- service-edge-router-policies (authorizes services to use edge routers)
You MUST have at least one of each but you can have many if you want. I would suggest you make an #all/#all service edge router policy (the tab you're on). Allow all services to use all edge routers.
Yeah Its Zac console running in docker.
Will try the #all thing next when I get back.
Thanks!
its zac console as a docker container but its on the same host as the controller but they are different containers on that host, but zac just points at that controller anyway.
Nearly there now, got the connection for ssh going over the link and its leaving the remote side, I added an iptables nat masquerade rule on egress, but now I see its getting connection reset, not sure if this is a local network problem but testing from the remote router itself, the connection works but from the netshoot container which is sitting on the local tunneller (sharing network ns) and testing it tries to connect but gets dropped after a few packets.
uh strange its sending down the loopback as the wrong source ip its sending the destination as the source like its source natting it or something. Is this an incorrectly configured host.v1 rule? 
recreated everything still some trouble, ssh seems to actually get through but snmp / udp 161 seems to not go anywhere it just dissappears into the overlay via Loopback?