Hi again, I have some questions regarding deployment of ziti routers / tunnels in a kuberenetes environment.
Currently have followed guides to deploy ztii router and tunnel using openziti helmcharts.
The tunnels run as deamonsets on every node.
Each node currently has a public external IP in order for the ziti tunnel to access the internet.
In each node we have a pod running a data-engine-app that can make outbound connections to applications host on different networks using ziti services.
Scenario:
Within one cluster Is it possible to:
host the tunnel/router in one public node.
host the data-engine-app in private nodes.
And then In the same k8 cluster:
could traffic be routed from the app in the private node through the ziti tunnel running in the public node to make requests to ziti services existing in different networks
Basically I'm under the assumption that the ziti tunnel and routers must have internet access into order to register with the controller and join the mesh.
But I don't want to exposeall my nodes to the public internet.
So i'm thinking of this solution in order to take advantage of ziti while still having the main apps in private nodes.
I think at the end of the day there are some important truths:
ALL OpenZiti components need to be able to make a connection to the OpenZiti controller, if you want access to services from "anywhere", the controller should be deployed with a public ip on the internet.
ALL OpenZiti overlay networks will need at least one router in the same address space of the controller, meaning if you want access to services from "anywhere", at least one router will need to be deployed with a public ip on the internet.
ALL other OpenZiti equipment can be deployed in private address space, with private ips and with DENY ALL INBOUND firewall rules.
In your example. I would deploy one router on the internet with a public IP and with link listeners enabled and I would deploy one router in the private network 1 and one router in the private network 2 with link dialers (no link listeners).
Then when private network 1 router comes online, it will form a link to the public router and when private network 2 router comes online it will form a link to the public router. This will create a mesh. Looking at your diagram, you could choose to not run a router in private network 2 as well and just run a tunneler, that's fine too.
Does this help? To me, kubernetes isn't relevant to this topology, it's a means to an end. It might end up making things easier or more complicated, but understanding that there needs to be a router "in the public" and a controller "in the public" AND NOTHING ELSE i think is the key concept here?
Sorry for the cross post with this thread. linking if somones interested in the future.
ALL other OpenZiti equipment can be deployed in private address space, with private ips and with DENY ALL INBOUND firewall rules.
controller "in the public"
at least one router in the same address space of the controller "in the public"
"in the public" means, has public routable address/ip and allows inbound connections
other routers or tunnels don't need to allow inbound connectoins, but do need outbound connections
This so far makes sense to me.
I guess my long shot question is if its feasible to only host the tunnel / router in the public subnet 2 . And somehow intercept/route traffic from the private subnet 2 through the router/tunnel in public subnet 2.
I'm not sure if my question makes complete sense. Or if this is even in the scope of what ziti can do.
Perhaps this is a case for using some sort of proxy to control traffic from private nodes to go through public node so that it can join the ziti network through the tunnel/router running there. Any clue if such a pattern like this make sense? Has there been such a pattern like this?
This sounds to me like you're looking for these? The first one has a link to a YouTube video where @rcsoleng walks through the steps... it sounds like what you're after? You'd have to make a route between the subnets though, right? I'm not a route-expert so, at this point i'll just leave these links and say "I gave it all I had".. hope this helps
The problem of deploying ZET on one node and trying to intercept traffic from another node by that ZET is difficult without configuring static routes outside of the nodes to route traffic between nodes. ZET uses tun interface to intercept packets that are forwarded to it by routing rules, so it is deployed as a daemonset with the host network mode enabled. Thus, the packets are intercepted locally on each node.
Edge Router has proxy and tproxy mode. The tproxy mode is similar to ZET Tun mode, but it uses iptables tproxy rules to intercept with ingress filtering steering. You may want to look into ER Proxy mode and see if that fits your requirement. Link to helm charts settings helm-charts/charts/ziti-router/values.yaml at 552fe38eb9465895f34fa4cfb2bd67a9213b1581 · openziti/helm-charts · GitHub and the link to guide Install the Router in Kubernetes | OpenZiti
2 Scenarios. Keep private nodes subnet private (e.g. AWS k8s) but need a replacement for Nat gateway via a ziti network (tunnel/router) to connect to a say DataBase in another ziti network (e.g. GCP) - Scenario 1 and Connect to a public internet url (Scenario 2)
Thank you for clarifying you're referring to EC2 public and private subnets, and the private subnet has a route to the internet via a NAT gateway.
Your EKS worker nodes occupy an EC2 private subnet, and you want pods scheduled on those nodes to be able to reach a remote service via Ziti, e.g., a database server.
First, do you wish to grant access to the Ziti service at the pod, node, or cluster level? The daemonset grants access on a per-node basis. The sidecar proxy is per-pod, meaning you can have different Ziti services granted to each pod on the same node. If you grant access to the Ziti service at the cluster level then all pods on all worker nodes will have access to the same Ziti services, unless you restrict access within the cluster at the network layer, e.g., with a NetworkPolicy.
Yes. Here's a sketch, though you might want something slightly different.
network 3 - ziti controller and router are securely reachable via the internet from networks 1 and 2
network 1 - running a ziti tunneler to host a ziti service for the database - has outbound access to the internet to reach the ziti controller and router
network 2 - private subnet w/ EKS worker nodes needs NAT gateway to translate private IPs via IGW for outbound internet access - nodes are not exposed to the internet
network 2 - public subnet w/ k8s controller nodes has direct, bi-directional internet access via IGW (nodes have public IPs, protected by EC2 security groups)
network 2 - install a Ziti tunneler at pod, node, or cluster level with client dial permission for the database service
@qrkourier to confirm: in the above : Nat gateway is not used for data transfer? One key purpose is to avoid the data processing/data transfer charges of using Nat gateway. If Nat gateway is used for translation only but the actual data comes via different mechanism...that is ideal (Different mechanism meaning > ziti router)
Hi thanks for the repsonse!
I've ammended the diagram to include your input and input from lumberjack.
network 3
hosts ziti controller and router that are accesisble through public internet
network 1
hosts database
host a router that has access to the database
host a router that can make outbound internet connections to controller and router in network 3
network 2
router in node in public subnet,
can make outbound internet requests to controller and router in network 3 through IGW
tunnel (deamonset) in node in private subnet,
can make outbound requests to controller and router in network 3 through NAT->IGW
When the data-engine-app in the private subnet makes a request to access the database using a ziti service url it gets intercepted by the ziti tunnel.
Does the tunnel forward the traffic immediately to the router in the public subnet without using the NAT, since they are in the same VPC?
I was hoping/wondering if the data transfer path for such a request would be in the blue line in the diagram. Almost like using ziti router in the public subnet as a http proxy (simlar to squid) for the ziti tunnel running in the private node.
Hope this makes it more clear!
Yes. I'm thankful for this clarification. I have some solid information for you this time!
Precisely.
Yes, if you do two things. Let's call network 2's ziti router in the public subnet "router2," and network 3's ziti router "router3." I assume router2 will receive connections from the tunneler via the private subnet's public SNAT IP address.
Configure router2 with an "edge listener" reachable by the worker nodes in the private subnet of that same VPC. For example, if the router's public IP can be resolved by domain name router2.ziti.example.com, then router2's config.yml has a section like this.
Grant permission for the ziti tunneler(s) in the private subnet to use router2. This is accomplished with an "edge router policy" (ERP). ERPs permit identities to use routers. If router2 is the first to answer the tunneler's request for an edge listener, it will be used. It will usually be first because it is closest, but if it's super busy then router3 might be used instead, which is probably a good thing.
Yes, it will follow the blue line if you configure a router-to-router "link" between networks 1 and 2. That is, if either router1 or router2 has a reachable public IP and is configured with a "link listener," then the ziti mesh fabric will have a direct link between networks 1 and 2, so it will usually be unnecessary for ziti's smart routing algorithm to relay via network 3, and only control plane data will flow to the ziti controller in network 3 as long as the direct router link is available.
For example, assuming router1 can "dial" router2.ziti.example.com:3022, and router2 has the following configuration, the direct router-to-router fabric transport "link" will be available.
link:
listeners:
- binding: transport
bind: tls:0.0.0.0:3022
advertise: router2.ziti.example.com:3022
dialers:
- binding: transport
You may wonder how the router can serve two protocols on 3022/TCP. Ziti uses ALPN and SNI to select the TLS servers for incoming ClientHello greetings on any configured port.
Imho, you are complicating your life with this public and private node set up as far as ziti is concerned. You can put entire cluster k8s in Network 2 behind NAT, so the cluster has outbound reachability to the internet. You can deploy edge router with the edge listener that is only reachable within the cluster service as described by @qrkourier . Then ziti-edge-tunnel can reach this router using that cluster service and data engine app would be able to reach anything across the ziti network through the ziti-edge-tunnel. data engine app --> zet --> er2 <--> er3 <--> er1 --> db. Also ZET can talk to er3 directly as well without even needing ER2
I assumed the public subnet was for K8S controller nodes serving the API server, possibly Ingress/Gateway workers, e.g., like a DMZ for things that need to be reachable on public IPs. Still, there may be a way to offload those public-facing cluster services with EKS's integration with AWS. I don't know enough about how it works.
Your point stands: You could publish any cluster services, including the K8S API server, as a Ziti service...if it doesn't need to be public.
Network 2's router doesn't need to be public, but they want a direct router link between networks 1 and 2, so at least one of router1, router2 needs to advertise a domain name resolving to a public IP w/ link listener.
data engine app would be able to reach anything across the ziti network through the ziti-edge-tunnel. data engine app --> zet --> er2 <--> er3 <--> er1 --> db.
In this case, wouldn't all the traffic between the data-enigne-app and the db traverse through the NAT? I know it may be complicated, but I'm trying to avoid using the NAT for data transfer to lower costs. The diagram above is trying to use a combination of the tunnel in private subnet and router in public subnet to make data transfer go through IGW instead of NAT.
Yes it will. I mentioned NAT bc it is just straightforward to block ingress traffic, but you can use security groups,fw, etc. If you want to use public subnet and the public IP NAT'ed to the node, then this does not change the fact that the router edge can be privately advertised to the cluster as a service and any Ziti Endpoints deployed in it can connect to it.