how'd you install and start the router? it's probably running with systemd? journalctl should get them for you:
journalctl -u ziti-router --since "10 minutes ago"
how'd you install and start the router? it's probably running with systemd? journalctl should get them for you:
journalctl -u ziti-router --since "10 minutes ago"
Ahh ok.
I'm starting it like this at the moment
root@ziti-router-colo1:~# /opt/openziti/bin/ziti router run /var/lib/ziti-router/config.yml
[ 0.013] INFO ziti/ziti/router.run: {arch=[amd64] routerId=[2z1SU5ph9] build-date=[2024-10-02T12:59:41Z] revision=[0eec47ce3c80] configFile=[/var/lib/ziti-router/config.yml] version=[v1.1.15] go-version=[go1.23.1] os=[linux]} starting ziti router
[ 0.014] INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[0] maxWorkers=[32] idleTime=[30s] poolType=[pool.link.dialer] maxQueueSize=[1000]} starting goroutine pool
[ 0.015] INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[128] idleTime=[30s] poolType=[pool.route.handler] minWorkers=[0] maxQueueSize=[1000]} starting goroutine pool
[ 0.015] INFO ziti/router/forwarder.(*Faulter).run: started
[ 0.015] INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[50] poolType=[pool.terminator_validation] idleTime=[30s] minWorkers=[0] maxQueueSize=[1]} starting goroutine pool
[ 0.016] INFO ziti/router/forwarder.(*Scanner).run: started
[ 0.016] INFO ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap: cached data model file set to: /var/lib/ziti-router/config.yml.json.gzip
[ 0.016] WARNING ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[ 0.017] INFO ziti/common.syncAllSubscribersEvent.process: {subs=[0]} sync all subscribers
[ 0.018] INFO ziti/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":5000000000,"DelayRxStart":false,"WriteTimeout":0,"MessageStrategy":null}
[ 0.018] INFO ziti/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"IntervalAgeThreshold":0,"MessageQueueSize":10}
[ 0.019] INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.rate_limiter] maxWorkers=[15] idleTime=[30s] maxQueueSize=[5000] minWorkers=[0]} starting goroutine pool
[ 0.019] INFO ziti/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[ 0.019] INFO ziti/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[ 0.019] INFO ziti/router/xgress_edge.(*Factory).CreateListener: xgress edge listener options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
lookupApiSessionTimeout=5s
lookupSessionTimeout=5s
channel.outQueueSize=4
channel.connectTimeout=5s
channel.maxOutstandingConnects=16
channel.maxQueuedConnects=1
[ 0.020] INFO ziti/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:9442]} starting channel listener
[ 0.020] INFO ziti/router/xgress_edge.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[10s] maxQueueSize=[1] poolType=[pool.listener.xgress_edge] minWorkers=[1] maxWorkers=[16]} starting goroutine pool
[ 0.021] INFO ziti/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:9442]
[ 0.021] INFO ziti/router/xgress_edge_tunnel.(*Factory).CreateListener: xgress edge tunnel listener options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
[ 0.023] INFO ziti/router/xgress_edge.(*Acceptor).Run: starting
[ 0.023] INFO ziti/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[ 0.023] INFO ziti/router.(*Router).getInitialCtrlEndpoints: controller endpoints file [/var/lib/ziti-router/endpoints] doesn't exist. Using initial endpoints from config
[ 0.023] INFO ziti/router.(*Router).startControlPlane: router configured with 1 controller endpoints
[ 0.024] INFO ziti/router/env.(*networkControllers).UpdateControllerEndpoints: {endpoint=[map[tls:secure.intern-dehogabw.de:9440:{}]]} adding new ctrl endpoint
[ 0.024] INFO ziti/router/env.(*networkControllers).connectToControllerWithBackoff: {endpoint=[tls:secure.intern-dehogabw.de:9440]} starting connection attempts
[ 0.099] INFO ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func3: {endpoint=[tls:secure.intern-dehogabw.de:9440]} successfully connected to controller
[ 0.099] INFO ziti/router/link.(*linkRegistryImpl).NotifyOfReconnect: {ctrlId=[NetFoundry Inc. Client Y786u1WKc]} resending link states after reconnect
[ 0.099] INFO ziti/router/xgress_edge.(*Factory).NotifyOfReconnect: control channel reconnected, re-establishing hosted services
[ 0.099] INFO ziti/router/xgress_edge_tunnel.(*Factory).NotifyOfReconnect: control channel reconnected, re-establishing hosted services
[ 0.103] INFO ziti/router/link.(*linkDestUpdate).ApplyListenerChanges: {routerId=[ad9eoEJBm] address=[tls:secure.intern-dehogabw.de:9442] linkKey=[default->tls:ad9eoEJBm->default]} new potential link
[ 0.103] INFO ziti/router/link.(*linkState).updateStatus: {key=[default->tls:ad9eoEJBm->default] oldState=[pending] newState=[dialing] linkId=[5tTb8tj6MxpD49STvptBTY] iteration=[0]} status updated
[ 0.103] INFO ziti/router/link.(*linkRegistryImpl).evaluateLinkState: {key=[default->tls:ad9eoEJBm->default] linkId=[5tTb8tj6MxpD49STvptBTY] iteration=[1]} queuing link to dial
[ 0.103] INFO ziti/router/link.(*linkRegistryImpl).evaluateLinkState.func1: {linkId=[5tTb8tj6MxpD49STvptBTY] iteration=[1] key=[default->tls:ad9eoEJBm->default]} dialing link
[ 0.103] INFO ziti/router/xlink_transport.(*dialer).dialSplit: {linkId=[5tTb8tj6MxpD49STvptBTY] connId=[4317ff1d-a020-4ef0-a14d-2693a918c4b6]} dialing link with split payload/ack channels
[ 0.103] INFO ziti/router/xlink_transport.(*dialer).dialSplit: {linkId=[5tTb8tj6MxpD49STvptBTY] connId=[4317ff1d-a020-4ef0-a14d-2693a918c4b6]} dialing payload channel
[ 0.105] INFO ziti/router/xgress_edge_tunnel.(*Factory).CreateDialer: xgress edge tunnel dialer options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
[ 0.105] INFO ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1 [ch{ctrl}->u{reconnecting}->i{75Zp}]: {terminatorId=[3koy8XfgLLEExRjpdLsvYb]} validating terminator
[ 0.106] INFO ziti/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1: received server hello, replying
[ 0.110] INFO ziti/router/state.(*apiSessionAddedHandler).instantSync: {strategy=[instant]} first api session syncId [cm2w4mxr780eaobpqsf4fam40], starting
[ 0.110] INFO ziti/router/state.(*apiSessionSyncTracker).Add: received api session sync chunk 0, isLast=true
[ 0.162] INFO ziti/router/handler_link.(*bindHandler).BindChannel: {dialed=[true] linkId=[5tTb8tj6MxpD49STvptBTY] routerId=[ad9eoEJBm] routerVersion=[v1.1.15] iteration=[1]} link destination support heartbeats
[ 0.162] INFO ziti/router/xlink_transport.(*dialer).dialSplit: {linkId=[5tTb8tj6MxpD49STvptBTY] connId=[4317ff1d-a020-4ef0-a14d-2693a918c4b6]} dialing ack channel
[ 0.175] INFO ziti/router/state.(*ManagerImpl).StartHeartbeat: heartbeat starting
[ 0.175] INFO ziti/router/xgress_edge_tunnel.(*tunneler).Start: {mode=[host]} creating interceptor
[ 0.175] INFO ziti/router/xgress_edge.(*CertExpirationChecker).Run: waiting 8541h50m4.876080481s to renew certificates
[ 0.206] INFO ziti/tunnel/dns.flushDnsCaches: dns caches flushed
[ 0.206] INFO ziti/tunnel/dns.NewDnsServer: starting dns server...
[ 0.224] INFO ziti/router/handler_link.(*bindHandler).BindChannel: {iteration=[1] dialed=[true] linkId=[5tTb8tj6MxpD49STvptBTY] routerId=[ad9eoEJBm] routerVersion=[v1.1.15]} link destination support heartbeats
[ 0.224] INFO ziti/router.(*xlinkAccepter).Accept: {iteration=[1] destId=[ad9eoEJBm] linkId=[5tTb8tj6MxpD49STvptBTY] dialed=[true]} accepted new link
[ 0.225] INFO ziti/router/link.(*linkRegistryImpl).applyLink: {newLinkIteration=[1] linkProtocol=[tls] newLinkId=[5tTb8tj6MxpD49STvptBTY] dest=[ad9eoEJBm] dialed=[true]} link registered
[ 0.225] INFO ziti/router/link.(*linkState).updateStatus: {key=[default->tls:ad9eoEJBm->default] oldState=[dialing] newState=[established] linkId=[5tTb8tj6MxpD49STvptBTY] iteration=[1]} status updated
[ 0.225] INFO ziti/router/link.(*linkRegistryImpl).notifyControllersOfLinks: {op=[link-notify]} attempting to queue link notifications
[ 0.225] INFO ziti/router/link.(*linkRegistryImpl).notifyControllersOfLinks.func1: {op=[link-notify]} link notifications starting
[ 0.225] INFO ziti/router/link.(*linkRegistryImpl).sendNewLinks: {ctrlId=[NetFoundry Inc. Client Y786u1WKc] op=[link-notify] iteration=[1] linkId=[5tTb8tj6MxpD49STvptBTY]} notified controller of new link
[ 0.225] INFO ziti/router/link.(*linkRegistryImpl).notifyControllersOfLinks.func1.1: {op=[link-notify]} link notifications exiting
[ 1.111] INFO ziti/router/state.(*apiSessionAddedHandler).applySync: finished synchronizing api sessions [count: 6, syncId: cm2w4mxr780eaobpqsf4fam40, duration: 102.698µs]
[ 2.207] INFO ziti/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
[ 2.207] INFO ziti/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
[ 2.211] ERROR ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.53:53: no such host
ziti-tunnel runs an internal DNS server which must be first in the host's
resolver configuration. On systems that use NetManager/dhclient, this can
be achieved by adding the following to /etc/dhcp/dhclient.conf:
prepend domain-name-servers 127.0.0.1:53;
[ 2.212] INFO ziti/tunnel/intercept.SetDnsInterceptIpRange: dns intercept IP range: 100.64.0.1 - 100.127.255.255
[ 2.217] INFO ziti/tunnel/intercept.(*ServiceListener).HandleServicesChange: {service=[service_desktop-365]} adding service
[ 2.217] INFO ziti/tunnel/intercept.(*ServiceListener).addService: {serviceId=[3tQqT76DEGDojE8tytQQ6U] serviceName=[service_desktop-365]} Hosting newly available service
[ 2.217] INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).establishTerminatorWithRetry.func1: {service=[service_desktop-365]} attempting to establish terminator
[ 2.223] INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse: {sessionId=[cm2w2ub6p7y1zobpq78zdrjhb] routerId=[2z1SU5ph9] terminatorId=[7gS3DUTIcAPeMvID81s8Fs]} received new session
[ 2.223] INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse: {routerId=[2z1SU5ph9] terminatorId=[7gS3DUTIcAPeMvID81s8Fs] createDuration=[5.353488ms]} received terminator created notification
[ 20.417] INFO ziti/router/xgress_edge_tunnel.(*Factory).CreateDialer: xgress edge tunnel dialer options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
[ 20.417] INFO ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1 [ch{ctrl}->u{reconnecting}->i{75Zp}]: {terminatorId=[7gS3DUTIcAPeMvID81s8Fs]} validating terminator
that comes here when I want to establish a connection:
[ 56.815] INFO ziti/router/xgress_edge_tunnel. (*Factory). CreateDialer: xgress edge tunnel dialer options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
[ 81,075] WARNING ziti/router/xgress. (*Xgress).rx [{c/xcwvq6UkB|@/MnBK}]: read failed (read tcp 192.168.100.15:54758->192.168.100.70:3389: use of closed network connection)
I edited your other post to use triple backticks -- it makes text and code easier to deal with.
if you could add fences to code/logs like this it'd be great, thanks. They go at the top and bottom of the text:
```
like this
```
that looks to me like it was not able to dial that port but it tried to. You're sure the port is 3389 and that port is open right? This one? 192.168.100.70:3389
I just looked at my logs and i get the same message in the router when connecting. I'm not sure why but when I see that message i am able to get a login prompt. Are you sure the RDP server/windows auth isn't getting in the way now?
Excuse me, I'll try to format it better in the future.
Now we are back at the beginning.
My configuration seems to be fine.
As already mentioned, I can connect a single RDP server without any problems.
Openziti doesn't seem to work with an "MS Remote Desktop Services Farm".
It's probably Microsoft's fault.
Briefly explained again.
The RDP client "desktop-365.domain.local" initiates the connection via the DNS of the farm.
It checks which server is free and then forwards it, e.g. rds1.domain.local.
This doesn't seem to work.
With openvpn it works without any problems, you can't compare it either.
I just wanted to say that it works in general
I still thank you for your patience and help
I just tested it again.
With the same configuration, a connection to an RDP server works fine.
The only difference is that it is not an RDS session host or a member of an RDS collection.
However, I have to enter the address and hostname in /etc/hosts on the router.
I did the same with the others and it seems logical to me.
That's really interesting. I'm sorry, I lost sight of the fact that you said it worked for RDP but NOT for RDS. I didn't understand the nuance of that -- I understand it better now, thank you.
It really does surprise me that this doesn't work with OpenZiti. OpenZiti's router will be dialing the same address as was intercepted so it will create a DNS request from the router for that address. It should be working exactly the same as Open VPN does. As to WHY it doesn't -- geesh that's a tough one.
Is it possible that through the RDS negotiation, the port changes somehow? For example with FTP, there are "active" and "passive" ports and the port switching can cause problems because OpenZiti is predicated on least privilege... Ports need to be explicitly mapped, could it be that the negotiation changes the ports in some way and that's causing a problem?
I'll ask @emoscardini if he's ever used RDS or not - he's got a bit more experience with some of these Windows administration type stuff - he might "just know". I'll see if I can learn anything about RDS vs RDP but unfortunately, since I don't have an RDS instance to test with, I'm not sure I'm going to be able to help.
Can you capture the packets from the router during one of these RDS->RDP sessions and see what sort of traffic shapes there are? Maybe you could share that with me via direct message or email (if you feel comfortable with that) or even use a zrok.io share
I haven't used RDS ever. I'll try to setup a lab this weekend, but in the meantime I've seen RDS farm may return to the client either the hostname or IP.
So is it possible to add to the host and intercept config the network 192.168.100.0/24?
And then try again?
I tested RDS with Browzer and that should work. It is web client based RDP though. Set up the Remote Desktop web client for your users | Microsoft Learn
BrowZer | OpenZiti
Many thanks to everyone for the great support.
I had already tried adding the networks 192.168.100.0/24 once.
But I will try again with the existing configuration.
I've already tried so many things that I've lost track.
The idea with the packet capture is very good, I could have thought of it myself.
It may be interesting to note that the RDP client reports when the connection is established:
Then nothing happens.
The RDS broker probably reports back to which server the connection should be established, which then does not work.
I didn't actually want to use the web client, probably habit.
But I'll have another look at it.
Thanks for the tip.
It would be worthwhile to do the packet capture on the client and also on the edge router at the same time. If the negotiation ends up informing the client to dial the IP, that would definitely explain why it failed if the IP wasn't also intercepted by the OpenZiti service.
Also if you have both captures, we might be able to inspect them on our side too. Packet capture and logs from both sides usually can tell the full story.
I will do it this morning and send it to them personally
Hooray I can't believe it, it works.
That was the right tip @natashell
Add the network 192.168.100.0/24 in the host.v1 and in the intercept.v1.
Should I post a summarized configuration if anyone else wants to implement this?
Thank you all, especially @TheLumberjack
Openziti is great
Amazing that's great to hear! Yes please as much data as you can and are willing to share is great! If you have a blogging media, a blog would be even better telling the world how you did it? If not, a post on Reddit will help spread the word is OpenZiti.
So glad to hear you got it all working!
Looking for a Zero Trust Network Access (ZTNA) solution, I came across OpenZiti. My goal was to allow laptops outside the corporate network to access a Microsoft "Remote Desktop Farm" – essentially, a VPN replacement. The main reason: improving security with a Zero Trust approach.
Although there were some challenges, the learning curve was steep. Once you understand the product, it’s not as difficult as it first seems. Here, I’m sharing my configuration – a thank-you to the helpful OpenZiti community and maintainers.
Note: This guide provides a rough setup overview. For detailed information, refer to the OpenZiti documentation.
curl -sS https://get.openziti.io/install.bash | sudo bash -s openziti-controller
Then run:
sudo apt install openziti-console openziti-router
sudo /opt/openziti/etc/controller/bootstrap.bash
Enter the following values (alternatively, add them to .env
in the same directory):
ZITI_CTRL_ADVERTISED_ADDRESS='xxxxxxxxx.xxxxx.de'
ZITI_CTRL_ADVERTISED_PORT='8440'
ZITI_USER='admin'
ZITI_PWD='strongPassword'
Enable the Controller service:
systemctl enable --now ziti-controller.service
Check if the service is running:
systemctl status ziti-controller
ss -tlnp | grep 8440
Adjust configuration if necessary:
File: /var/lib/ziti-controller/config.yml
Restart the service:
systemctl restart ziti-controller.service
Check the logs:
journalctl -u ziti-controller --since "10 minutes ago"
Some steps can also be done via the ZAC: https://xxxxx.xxxxx.de:8440/zac
ziti edge login xxxxxxx.xxxxx.de:8440 -u admin -p strongPassword
Create an Edge Router:
ziti edge create edge-router public-router --jwt-output-file public-router.jwt
Run /opt/openziti/etc/router/bootstrap.bash
and adjust the bootstrap.env
file if necessary.
Troubleshooting: In the generated /var/lib/private/config.yml
file, modify this line:
cert: "router.cert"
/var/lib/private/ziti-router/router.cert
If token errors occur, run manually:
ziti router enroll /var/lib/private/ziti-router/config.yml --jwt /var/lib/private/ziti-router/pub-er.jwt
Enable and check the service:
systemctl enable --now ziti-router.service
systemctl status ziti-router.service
curl -sS https://get.openziti.io/install.bash | sudo bash -s openziti-router
Login to the Controller:
ziti edge login xxxxxxx.xxxxx.de:8440 -u admin -p strongPassword
Create the Edge Router:
ziti edge create edge-router "private-router" --jwt-output-file private-router.jwt --tunneler-enabled
Configuration: Run /opt/openziti/etc/router/bootstrap.bash
and adjust the bootstrap.env
file if necessary.
Troubleshooting (as above):
ziti router enroll /var/lib/private/ziti-router/config.yml --jwt /var/lib/private/ziti-router/private-router.jwt
.jwt
file via "ADD IDENTITY".Access Configuration (intercept.v1):
*.domain.local
Hosting Configuration (host.v1):
private-router
).*.domain.local
192.168.100.0/24
.192.168.100.0/24
.If necessary, add entries in /etc/hosts
:
192.168.100.70 farm-collection-name.domain.local rds-broker.domain.local
192.168.100.71 rds1.domain.local
192.168.100.72 rds2.domain.local
The RDP client connects via farm-collection-name.domain.local
, automatically assigning to the appropriate session host based on load.
I hope I didn’t miss anything or mix anything up. Suggestions for improvement are welcome!
That's great! Thanks so much for the writeup! If you promote it anywhere else, let us know and I'll help retweet or promote it however I can. Cheers!