Questions about latency and routing

Ziti Overlay
1

Hi,


I am in the process of setting up an overlay network for my homelab. I have some questions regarding latency and the routing behavior of OpenZiti.

I'll first start by sketching my current setup:

Site A:

  • VM1 that is running a Minecraft server (in Docker) and the ziti-edge-tunnel Linux package.
  • VM2 that is running a public Ziti edge router (tunneler) in Docker (r2).

Site B:

  • My dekstop that is running the Ziti Desktop Edge for Windows.

Site C:

  • VM3 that is running the Ziti controller and another public edge router (tunneler) (r1).

Site A and B are ~5km apart. Site C is ~350km away from the other two sites.

My Ziti config:

  • There is an edge router policy that allows #all identities to use #all routers.
  • There is a service edge router policy that allows #all services to use #all routers.
  • I have defined a simple service (mc1.ziti) that allows my @Desktop to connect to the Minecraft server over the ziti-edge-tunnel on @VM1.
  • I have defined another simple service (mc2.ziti) that allows my @Desktop to connect to the Minecraft server over the @r2 router.

Both services work. Only the latency is not really ideal, and the routing does not work how I expect it to. Initially, I started off with just r1 and the ziti-edge-tunnel on VM1. When joining the Minecraft server though, I had an average latency of ~110ms. I found this strange, because I also have a Wireguard tunnel between VM3 and VM2, and use VM3 as a reverse proxy. Over the Wireguard tunnel, my latency is ~52ms.
Thinking it was related to the physical distance between the machines, I set up r2 in the same underlay network as VM1. Over r2, the average latency was ~52ms, which I still find high, considering that my latency over a direct underlay connection is ~28ms. Also, while the average latency is technically playable, there were still peaks of >100ms (albeit not that noticable).

One other thing that I don't really understand, is the routing. Sometimes, on mc2.ziti I am connected directly through r2. Other times, ziti sends me over r1 -> the link -> r2. Also, for mc1.ziti, it seemingly never picks r2 to reach the tunnel on VM1.
And, to make matters even more confusing, When I am connected on mc2.ziti over r1 -> the link -> r2 (which covers the same distance as mc1.ziti over r1), the average latency is lower (~70ms).

Long story short:

  • Is this amount of latency expected?
  • Does the physical distance to the controller affect the latency of the data flow?
  • Mesh links have the latency to each other. Is this latency not taken into account for the routing?
  • Is this routing behavior expected?

Some bonus questions, while I am here :grin::

  • I am using host mode on both my edge routers. When would I have a need for the tproxy mode?
  • What is the use of private edge routers? Is it for HA?
  • In the docs, and in the videos, there are seemingly different types of identities. But, I can't seem to find how to set the type, or what the impact is of the type (if any).
  • What is the impact of the admin attribute on an identity? I first started my identities of with that on, but I could not access any services (even though they appeared in the desktop client).

The latency results (not accurately measured, just kind of eyeballed):

Connection type Physical distance (~) Avg. latency (ms) Peak latency (ms)
Direct underlay connection 5km 28 50
Remote hosted reverse proxy -> Wireguard tunnel 350km 52 70
Direct connection to r2 (mc2.ziti) 5km 52 100
r1 -> the link -> r2 (mc2.ziti) 350km 70 150
r1 -> ziti-edge-tunnel on VM1 (mc1.ziti) 350km 110 160

Apologies for the long post,
Glenn

2

Hi @gdcooman, welcome to the community and to OpenZiti!

I've run Minecraft over OpenZiti a fair bit, often using zrok itself but also on its own and I personally never had a problem with any latency, so I never ended up measuring it! Anecdotally, I just remember it being smooth. With that out of the way... :slight_smile: I did fire up my own Minecraft server again (it's been a minute), turned on the Java version and measured my own personal latency. It was clocking in at over 100ms as well which confused me, so I started to dig...

You mentioned you have ~50ms of wireguard latency and about double that with OpenZiti and you ask if that's expected. With OpenZiti that's probably more inline with what I'd expect. Currently, OpenZiti works in a TURN-style. This means your traffic must traverse through a router to get to another identity where the traffic is offloaded. Wireguard on the other hand will allow you to punch through a firewall using a STUN server (assuming you're behind a firewall and it's needed, and it's supported/not blocked by the firewall). At that point, you're directly connecting from one location to another with no relay needed. So that hopefully explains the routing a bit.

As for distance, yes distance absolutely matters for latency as it relates to the speed of light not to mention all the other internet backbone/isp routers that the traffic needs to traverse. So more distance + more routers = more latency. No way around that.

Nominal Results

For me, I first started using a router in Arlington VA. That is about 400mi/650ki one way. I have my minecraft server running on a laptop in a closet, and i used this laptop as the client. When I connect through this router my latency measured by Minecraft's in-game F3+3 widget is ~160ms with bursts like you noticed (up to 248ms)

image
image497×154 14.8 KB

So I deployed a router in my house to take the internet and my ISP out of the equation. As you would expect, the speed is dramatically faster. It still spikes for reasons unknown, but much less and overall the latency is much faster. At times I could see 7ms reported (I've seen it as low as 5ms). Oh, I also went so far as to eliminate jitter caused from wi-fi and plugged these two laptops directly into a switch:

image
image492×148 15.3 KB

but I also saw lots of chuggy server/local machine + local network/pc related stuff and it would get into the teens 14-20ms ish:

image
image494×155 18.7 KB

Directly connecting to the same server, gets me about 5-6ms pretty consistently:

image
image496×151 15.7 KB

Overall, I would say OpenZiti's latency is negligable: 1ms maybe 2ms (if even that). It's much more important how far you are from the router and how far the offload point is from that same router.

Watch the Terminator Location

Now, one thing to be very aware of is the location of the 'terminator' for this service. While doing my testing, I had turned off my local router to make sure I was going out to Ashburn, etc. This ended up causing me a problem I was not aware of. Doing this triggers a known quirk with our existing routing implemenation that is something we plan to fix, just isn't fixed yet. When I turned off my local router, the terminator for the service (the last router, before traffic is sent to the end service) switched to the Asburn router, and would not switch back to the local router. This ended meaning traffic would route from my local minecraft zdew -> local rotuer -> ashburn -> minecraft server zdew and my latency was very bad.

You can check this by running ziti fabric list circuits (shown below using windows):

.\ziti.exe fabric list circuits
╭───────────┬───────────────────────────┬──────────────────────────┬────────────────────────┬─────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ID        │ CLIENT                    │ SERVICE                  │ TERMINATOR             │ CREATEDAT           │ PATH                                                                                                             │
├───────────┼───────────────────────────┼──────────────────────────┼────────────────────────┼─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Iuk-M2Ew0 │                           │ salt4506                 │ 6W8j9TVlte8fa3eGpzbDoA │ 2025-06-03 19:14:01 │ r/OCI us-ashburn-1 Edge Router 1 -> l/5k9r49WKgDur32wr2mnGhV -> r/nc-router-b4984c49-7b37-49d8-a3bd-2f440526c0a7 │
│ gxVXM2EP3 │ cmbgpem3m5v6yja05kt0luzy2 │ minecraft-java-parkplace │ 1L8x8xhfIUnZZ0W4Iyom8c │ 2025-06-03 19:13:59 │ r/dovanet-home-router                                                                                            │
│ s5jwouEP0 │                           │ salt4505                 │ 5IYegKS8JKGouEnDMFUQd8 │ 2025-06-03 16:11:29 │ r/OCI us-ashburn-1 Edge Router 1 -> l/5k9r49WKgDur32wr2mnGhV -> r/nc-router-b4984c49-7b37-49d8-a3bd-2f440526c0a7 │
╰───────────┴───────────────────────────┴──────────────────────────┴────────────────────────┴─────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
results: 1-3 of 3

You can see the 'salt' services, they are connecting through ashburn to get to my local router: r/OCI us-ashburn-1 Edge Router 1 -> l/5k9r49WKgDur32wr2mnGhV -> r/nc-router-b4984c49-7b37-49d8-a3bd-2f440526c0a7

But my Minecraft service is only traversing my local router r/dovanet-home-router

So, if you find you're getting really bad latency, more than you expect, try restaring your terminating client. This will force it to renegotiate the terminator to decrease your latency. Hopefully taht all makes sense???? :slight_smile:

Bonus Questions

trpoxy is only necessary if you plan to use that router to also intercept traffic. If you use it for offload only, there's no need for tproxy with what you're doing so far

"private" edge routers are ones deployed in "private networking space", that's all. They are exceptionally useful in the exact manner as I have deployed them, when you don't want to trust your local network or you want to short-circut going out to the internet and back again. For "home lab" people, probably not super useful since you'll probably 'trust' your home network. But for a true 'zero trust' setup, you'd want a local, "private" router. If you run the ziti cli command to generate the config, what you'll see is these routers don't advertise links, that's the only difference. They will dial other routers to form the mesh, but they just don't advertise links...

There are two: "identities" and "router" identities. You can only create "regular" identities. Router identities are managed by the overlay and they are generated when you enable tunneling mode on your router. I nearly always enable tunneling mode and use the identities to offload traffic (as you have)

It'll allow that identity to modify the overlay network. So it can create/delete/update services, policies etc. etc...

Ok. I hope that answers your questions. I've learned something today myself, so that's fun!

3

Hi, @TheLumberjack

Thank you very much for your answers!

While I understand that Wireguard and Ziti do not work in the same way, I don't understand however why the difference in my tests is that big.
With my Wireguard setup, the traffic goes from my client in site B, to the reverse proxy in site C, over the Wireguard tunnel to site A, and then back.
With the Ziti setup, the traffic goes from my desktop client in site B, to the router in site C, to the server in site A, and then back (unless I am misunderstanding something). So, the distance seems largely the same to me.

With regards to the distance, I was more asking if the distance to the controller matters a lot once the connection is established?

The quirk you mentioned, is that the reason that the routes Ziti picks in my first post are sometimes not the most 'logical'? Are the terminators only evaluated when the terminating client first starts? I did find that the terminators would switch, seemingly at random.

I am wondering though how you connect to a private router? How does the Ziti desktop client (or the controller) know that the private router is reachable? Also, how do I pass the --private flag to the router when running it in docker?

And lastly, is it normal that an identity with the admin attribute cannot access any services (that it has access to)?
I just tested it, and now it seems to work. I'll test some more before I come back to this.

Thanks,
Glenn