Hi,
I am in the process of setting up an overlay network for my homelab. I have some questions regarding latency and the routing behavior of OpenZiti.
I'll first start by sketching my current setup:
Site A:
VM1
that is running a Minecraft server (in Docker) and the ziti-edge-tunnel Linux package.VM2
that is running a public Ziti edge router (tunneler) in Docker (r2
).
Site B:
- My dekstop that is running the Ziti Desktop Edge for Windows.
Site C:
VM3
that is running the Ziti controller and another public edge router (tunneler) (r1
).
Site A and B are ~5km apart. Site C is ~350km away from the other two sites.
My Ziti config:
- There is an edge router policy that allows
#all
identities to use#all
routers. - There is a service edge router policy that allows
#all
services to use#all
routers. - I have defined a simple service (
mc1.ziti
) that allows my@Desktop
to connect to the Minecraft server over the ziti-edge-tunnel on@VM1
. - I have defined another simple service (
mc2.ziti
) that allows my@Desktop
to connect to the Minecraft server over the@r2
router.
Both services work. Only the latency is not really ideal, and the routing does not work how I expect it to. Initially, I started off with just r1
and the ziti-edge-tunnel on VM1
. When joining the Minecraft server though, I had an average latency of ~110ms. I found this strange, because I also have a Wireguard tunnel between VM3
and VM2
, and use VM3
as a reverse proxy. Over the Wireguard tunnel, my latency is ~52ms.
Thinking it was related to the physical distance between the machines, I set up r2
in the same underlay network as VM1
. Over r2
, the average latency was ~52ms, which I still find high, considering that my latency over a direct underlay connection is ~28ms. Also, while the average latency is technically playable, there were still peaks of >100ms (albeit not that noticable).
One other thing that I don't really understand, is the routing. Sometimes, on mc2.ziti
I am connected directly through r2
. Other times, ziti sends me over r1
-> the link -> r2
. Also, for mc1.ziti
, it seemingly never picks r2
to reach the tunnel on VM1
.
And, to make matters even more confusing, When I am connected on mc2.ziti
over r1
-> the link -> r2
(which covers the same distance as mc1.ziti
over r1
), the average latency is lower (~70ms).
Long story short:
- Is this amount of latency expected?
- Does the physical distance to the controller affect the latency of the data flow?
- Mesh links have the latency to each other. Is this latency not taken into account for the routing?
- Is this routing behavior expected?
Some bonus questions, while I am here :
- I am using host mode on both my edge routers. When would I have a need for the tproxy mode?
- What is the use of private edge routers? Is it for HA?
- In the docs, and in the videos, there are seemingly different types of identities. But, I can't seem to find how to set the type, or what the impact is of the type (if any).
- What is the impact of the admin attribute on an identity? I first started my identities of with that on, but I could not access any services (even though they appeared in the desktop client).
The latency results (not accurately measured, just kind of eyeballed):
Connection type | Physical distance (~) | Avg. latency (ms) | Peak latency (ms) |
---|---|---|---|
Direct underlay connection | 5km | 28 | 50 |
Remote hosted reverse proxy -> Wireguard tunnel | 350km | 52 | 70 |
Direct connection to r2 (mc2.ziti ) |
5km | 52 | 100 |
r1 -> the link -> r2 (mc2.ziti ) |
350km | 70 | 150 |
r1 -> ziti-edge-tunnel on VM1 (mc1.ziti ) |
350km | 110 | 160 |
Apologies for the long post,
Glenn