I am currently on designing an infrastructure in AWS with EKS and OpenZiti and I thinking on how to integrate OpenZiti with EKS Pods. I stumbled over your docs which are really helpful. I saw that there are several options on how to achieve an integration with EKS.
But, for our use-case we need zero-trust on certain pods (so zero-trust on pod-level) and therefore we would need to inject a sidecar proxy using the tproxy mode. (tcp proxy is not an option)
The downside of the tproxy side-car is that it needs the net_admin capability in a pod - which is tolerable for standard use-cases, but in our case it is not, as a user will have direct access to a shell in that pod and therefore capabilities need to be highly restricted.
For this reason I was thinking whether it is possible to write the ip-tables rules ( right now done by the side-car tproxy) via a CNI-Plugin, so that the pod itself would not need the net_admin capability.
From my current knowledge, this is also what service meshes like linkerd and istio are doing.
The plugin idea looks promising, and I'll need to catch up on how those work to say more.
Meanwhile, did you rule out the possibility of an opaque forward proxy port, or is that what you were referring to when you said TCP proxy isn't an option, e.g., 127.0.0.1:3000 where 3000/tcp is a TCP proxy port for a particular Ziti service provided by a sidecar running ziti tunnel proxy?
I understand the zero-trust constraint applies to the pod level, so a cluster-level proxy is not an option.
yes I meant that ziti tunnel proxy is not an option, as our application would need to know somehow, which port to speak to... - at least I cannot image how that could work.
Can you image any other solution we could go with, despite from the TCP proxy port ? (for the time being)
Probably by integrating a service mesh like istio or cilium ? (that's anyway something we are thinking about)
I don't see an alternative if NET_ADMIN can't be granted to the pod. As I understand, that's needed to install the TPROXY rules. I'll poke a couple of colleagues in case they have alternatives to offer. It makes sense to me for us to adopt the pattern used by Istio because their proxy sidecar serves a similar purpose, only the Ziti sidecar is intentionally split-tunnel, not requiring all destinations to be on-mesh like Istio's sidecar, with external exceptions only. Rather, Ziti's sidecar allows any destination and only "intercepts" Ziti service addresses.
I see that a hypothetical Ziti CNI could be chained to whatever existing CNI, e.g., Calico, Flannel. That's good news because I had believed the cluster admin had to choose exactly one CNI or use a more complicated multiplexing CNI framework.
That sounds promising!
So in case we have Cilium or Istio in place we could use the "ServiceMesh CNI plugin" and the hypothetical Ziti CNI plugin at the same time. Do you see a way of implementing such a Ziti CNI plugin for validating this expected behavior ?
I think that such a feature could greatly improve the interoperability with K8s
Ziti is a service mesh and one needs ziti router with edge/tunnel to be deployed in a cluster to enable that mesh between microservices in that cluster at the very least (similar to the ambient mesh) . We also have the ebpf interception now (more about it here https://github.com/netfoundry/zfw) that is available at the VM level at the moment. We are thinking about how to use it in k8s natively. The cni plugin looks interesting. Also, we are trying to gauge what is the percentage of k8s users that prefer node level proxy vs pod level as well.
Great to hear that you are thinking about these these things.
I can definitely say, that I would prefer to have a pod-level proxy, otherwise there wouldn't be an end-to-end trust between services - which is actually a major feature of OpenZiti.
Have you already tested the ebpf interception within K8s - meaning is there a repository available ?
Are you planning on implementing a CNI plugin for Ziti to test whether that could work ?
I'm asking, as there is currently no possibility for K8s pod egress traffic interception without granting additional capabilities to a pod - despite the LoopbackProxySidecar, which does not provide DNS.
Yes, we are thinking/brainstorming for sure and definitely encourage inputs/suggestions to see what make sense long term for egress side of things for most users. We have the ebpf code that can be deployed at the vm level. We have not started the work yet for containers. Most likely would be at the node level, since you need elevated permissions, i.e. CNI or CNI plugin. As you know yourself that CNIs like Cilium use ebpf too. We are not clear how it would work yet if we were to develop a plugin. You can run multiple ebpf programs in sequence today and there is also more work being done in that space to enhance that to mix and match ebpf programs. It may not be an issue at all. Here is the source code if you want to have a look or even contribute zfw/src.
One thing we always ask for the pod level proxy users, why not embed ziti listener/dialer into apps/microservices? By doing it, all the networking connectivity issues kind of go away.
Thanks for the reply and being open about the progress on this topic.
I'm definitely willing to contribute, as this is sth. me and my colleagues are dependent on
I will try to a push this and create a task for it in our Kanban - will come back to you as soon as I can work on it
