OpenZiti concept clarifications

[lzt]

Hello!

My team and I have been trying to understand some of the concepts that make up OpenZiti. We ended up having several questions based on some of our readings:

1. What is the difference between fabric and overlay in the Ziti context, if there is any?

2. In the guide here (Local - Docker Compose | OpenZiti), there is a representation of the network that is built. Do the red and blue networks represent two different overlay networks?

3. The guide states that the ziti-fabric-router-br is the only router that can communicate to all other routers, does this imply that the red network cannot see the blue network except via this specific router?

4. Is it possible to set up security policies between these networks? Would these be the equivalent of setting up identity, service, and edge router policies or just the latter mentioned?

5. Are there any differences in the type of routers deployed in Ziti or is everything more or less a edge router?

6. Do tunnelers spin up edge-routers on their own? Doing some light testing, we noticed that on the client and server machines where tunnelers were running, there were ip route entries for 100.64.x.x, which were the IPs assigned on the Ziti overlay (fabric?), but on the ziti controller/edge router machine, there was no such entry.

I appreciate the help received so far in this community and continuously looking forward to learning about this project.

[TheLumberjack]

Hi @lzt, let me try to answer these questions.

@lzt

What is the difference between fabric and overlay in the Ziti context, if there is any?

For most people they are the same. But from here on, this will be a "hair-splitting" type of reply. IN GENERAL, for people using OpenZiti I would say, "not really any difference". Now, let me go into details... "The Overlay" is a term we use for the OpenZiti overlay network. That encompasses all the parts of the network. The OpenZiti controller, the OpenZiti routers: Edge Routers, "fabric routers" etc. The fabric is usable by more than just OpenZiti. It was designed for extensibility, should anyone want a fantastic mesh network without the "extra" good stuff that is OpenZiti. So, OpenZiti is a fabric, but "the fabric" doesn't necessarilly mean OpenZiti. In practicality, this is so nuanced that at this time, and for most people it's not relevant. But it's not entirely the same thing. I hope that helps?

@lzt

In the guide here (Local - Docker Compose | OpenZiti), there is a representation of the network that is built. Do the red and blue networks represent two different overlay networks?

No. They are an attempt to isolate two "underlay" networks similar to having two separate VPC's in amazon or two different locations. It's an attempt to emulate two different networks which can be bridged using the OpenZiti overlay. So there are two sites, red and blue and "the internet" is represented as "red+blue" (purple). It's imperfect, but that's the idea...

@lzt

The guide states that the ziti-fabric-router-br is the only router that can communicate to all other routers, does this imply that the red network cannot see the blue network except via this specific router?

Yes. The edge router in the blue network (as it comes from the compose file), would not be able to send traffic to a node in the red network via the 'underlay', it can only do that via the overlay. That's exactly what it implies. It's not ONLY via that router - no. Using the overlay traffic could go from:
blue network client -> ziti-private-blue (blue network) -> ziti-edge-rotuer (internet/purple network) -> ziti-private-red (red network) -> red network server.

But really, it's up to 'the fabric' to route that traffic. It MIGHT go through the ziti-fabric-router-br, it might not! That's the beauty of the fabric, it figures out the best path.

@lzt

Is it possible to set up security policies between these networks? Would these be the equivalent of setting up identity, service, and edge router policies or just the latter mentioned?

security policies in the form of "edge router policies" and "service edge router policies"? Certainly. Any other policies and I'm not entirely sure I understand the question.

@lzt

Are there any differences in the type of routers deployed in Ziti or is everything more or less a edge router?

I strongly suggest you only deploy edge routers. You can decide if those edge routers are transit only (ie they don't accept edge connections). There's no downside imo to deployijng edge routers exclusively and I'd recommend you just do that.

@lzt

Do tunnelers spin up edge-routers on their own?

Edge routers can be tunneler-enabled. In that case they are tunnelers on their own. The quickstart always deploys them in "host" mode, meaning they are only for offload from the fabric but the edge routers also support a "tproxy" mode if you want to have the same sort of intercept capabilities as a normal tunneler. I'd say "usually" that's not how I've seen people deploy. For "intercept" type functionality it's usually easier/better to deploy one of the ziti-edge-tunneler-based tunnelers. Ziti Desktop Edge for Mac/Windows/linux.

@lzt

we noticed that on the client and server machines where tunnelers were running, there were ip route entries for 100.64.x.x, which were the IPs assigned on the Ziti overlay

This is how OpenZiti accomplishes the "intercepting" functionality. So if you have a tunneler running, yes you'll see those routes. The controller doesn't have a tunneler running so you won't see them there, and as mentioned above, usually people deploy edge routers with "host" mode enabled which doesn't need routes. (also though tproxy mode would end up using IP tables anyway, I think)

Hope that helps and makes sense. If not, let us know!

[lzt]

Thanks for the answers @TheLumberjack. I have a couple follow ups for the last two questions I had:

@TheLumberjack

This is how OpenZiti accomplishes the “intercepting” functionality. So if you have a tunneler running, yes you’ll see those routes. The controller doesn’t have a tunneler running so you won’t see them there, and as mentioned above, usually people deploy edge routers with “host” mode enabled which doesn’t need routes. (also though tproxy mode would end up using IP tables anyway, I think)

@TheLumberjack

Edge routers can be tunneler-enabled. In that case they are tunnelers on their own. The quickstart always deploys them in “host” mode, meaning they are only for offload from the fabric but the edge routers also support a “tproxy” mode if you want to have the same sort of intercept capabilities as a normal tunneler. I’d say “usually” that’s not how I’ve seen people deploy. For “intercept” type functionality it’s usually easier/better to deploy one of the ziti-edge-tunneler-based tunnelers. Ziti Desktop Edge for Mac/Windows/linux.

So in our setup when we had a controller/edge VM, a client w/ ziti-edge-tunneler, and a server w/ ziti-edge-tunneler, how is traffic being routed under the hood from:

client --> ziti controller/edge router vm --> server

if there is no route defined on the edge router machine? Is this done on the overlay level? If so how would that translate in the Linux routing table? Please feel free to correct me if there's any concept I am misunderstanding as well and this is not how things work. Thanks again!

[TheLumberjack]

@lzt

we had a controller/edge VM, a client w/ ziti-edge-tunneler, and a server w/ ziti-edge-tunneler, how is traffic being routed under the hood

Hi @lzt. Here's a common situation as you mention. A client using a web browser sends traffic to a secure web server over the OpenZiti overlay network.

I'm not a network engineer by trade but I think this is pretty accurate (or "close enough")... Here's the basic flow for a DNS-based interaction:

• Client starts and authenticates to the OpenZiti controller and gets authorized for a service providing HTTP access and configured with an "intercept.v1" config of "my.private.web.server"

• Client provisions an IP for the intercept on the DNS provided by the OpenZiti tunneler. "my.private.web.server"==100.64.0.30

  • NOTE: this is not a 'real' IP on the network, it's only valid in the scope of the OS itself.

• Client adds route to local system. All configured IP space should route to the TUN IP (usually 100.64.0.1)

• Client makes a request to "my.private.web.server".

• "my.private.web.server" is resolved using a DNS server provided by the tunneler (usually 100.64.0.2) and gets answer 100.64.0.30 (as an example, some IP in the configured space)

• browser/system sends traffic IP 100.64.0.30

• As mentioned, the system is configured with a route that routes all traffic in the configured space to the TUN IP. Example on windows:

  Network Destination        Netmask          Gateway       Interface  Metric
        100.64.0.0      255.192.0.0         On-link        100.64.0.1    511
  

• System routes traffic to 100.64.0.1

• OpenZiti tun receives traffic for 100.64.0.30. It then maps that IP to the proper service, wraps the packets with OpenZiti protocol, and puts the wrapped packs into the overlay for routing to the destination

• OpenZiti fabric routes traffic efficiently to the destination (the 'terminator')

• Traffic arrives at the destination

• Destination knows the packets are for "10.0.1.1" (the IP of the private web server) through the host.v1 config.

• Destination opens UDP/TCP connection to the destination, and proxies packets back/forth

Hows that - that make enough sense? (I hope)

[PhilipGriffiths]

@lzt

Are there any differences in the type of routers deployed in Ziti or is everything more or less a edge router?

To expand for understanding, all routers are the same base image, and you have the choice as to whether you turn on or off the 'link listener'.

• Edge Router, link listener turned off: ER can see all other ERs and forms outbound connections to ERs with LL turned on. This allows it to be in private address space.
• Edge Router, link listener turned on: ER can see all other ERs and forms connections to all other ERs with LL turned on. This must be in public address space.

Ziti uses a dijkstra's algorithm to calculate the lowest latency path available across the fabric. All this functionality enables the Ziti data plane to operate with greater resiliency and performance while keeping the edge dark and secure.