Docker Swarm overlay networks (VXLAN) using ziti underlay

Ziti Overlay
1

I have just stumbled upon openziti and a cursory glance of the docs makes it seem like it may be useful for my networking problem but I'm hoping for a more expert opinion on my specific scenario before I dive into the deep end.

I run a homelab with 10 nodes of varying x86/arm architecture but all my services are deployed using plain docker compose stacks, not swarm services. However, all my nodes use docker swarm mode, and are joined in a single cluster, so I can leverage docker overlay networks (vxlan) for the same reason anyone would use tailscale, netbird, openziti, etc...all of my inter-sevice routing, including reverse proxy, address and route traffic through the overlay networks rather than having to deal with per-host addressing.

I have recently spun up a VM/VPC with a cloud provider and would now like to have this VM host join my docker swarm so I can use overlay networking with services deployed there, as well. Keeping this exact network topology -- all hosts in my lab routing through docker overlay networks -- is a dealbreaker for me because I would like to keep deployment/routing transparent and network stack agnostic: If I move a service from VPC VM1 to Node3 (in my local network) I don't want to have to reconfigure networking for the compose stack or any other service that communicates with it.

This has been a nightmare to get working with all the Wireguard based zero trust solutions I've tried so far (netbird and pangolin). Normal host-to-host comms work fine but getting the host to join swarm and overlay networks has been difficult or impossible. Likely partially due to UDP usage with WG and docker overlay/ingress ports and (very probably) packet overhead caused by wireguard + vxlan conflicting with MTU settings for docker overlay/ingress interfaces.

Since openziti uses a wholly different approach for connectivity, which primarily relies on TCP, I have my hopes up that the above problems wouldn't be something I encounter when using it. But that is why I'm here looking for more insight. Does my scenario sound feasible, using openziti?

To summarize what I want to achieve:

  • Existing Docker Swarm on Private Network 1 with overlay networks (vxlan) networks
  • Openziti controller + router on VPC VM1, router on Host X within Private Network 1
  • VPC VM1 host joining Swarm on Private Network 1 via openziti router
    • Deploy docker services that communicate over docker overlay network

Thanks for your time.

2

Hi @FoxxMD, welcome to the community and to OpenZiti!

Personally, I know precious little about Docker swarm. I've seen the "overlay networks" in the past but I've never had the time to get around fiddling with them to learn about them.

This is something that OpenZiti generally excels at imo, but everyone's needs are different and what I/we think is elegant you might not find acceptable. This is a great goal though and totally understandable.

It's hard to know for sure since you're entering waters that I don't think anyone has waded into just yet. I know I haven't, so it's hard for me to really even know.

I did some chat GPT'ing. Docker and compose both allow you to set a different container as a container's network, effectively sidecar'ing the container. I was hoping you could do that with swarm but it seems like swarm doesn't support that. OpenZiti's routers do allow you to set them up as a gateway like is shown here Use a Router as a Local Gateway | OpenZiti Maybe that will help?

Sorry I can't be more help on this one. Cheers

3

Appreciate the response. I think the local gateway is as close to what I am looking for WRT joining the VM host transparently into the swarm. I'll give that a try and report back when I have some results.