# CNI Plugin for OpenZiti **URL:** \ **Category:** Support\ **Created:** [August 25, 2023, 9:44am UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552 "2023-08-25T09:44:57Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![janst](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/janst/32/3076_2.png) [@janst](https://openziti.discourse.group/u/janst)\ **Post date:** [August 25, 2023, 9:44am UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/1 "2023-08-25T09:44:57Z")
Hi there, I am currently on designing an infrastructure in AWS with EKS and OpenZiti and I thinking on how to integrate OpenZiti with EKS Pods. I stumbled over your [docs](https://openziti.io/docs/guides/kubernetes/workload-tunneling/) which are really helpful. I saw that there are several options on how to achieve an integration with EKS. But, for our use-case we need zero-trust on certain pods (so zero-trust on pod-level) and therefore we would need to inject a sidecar proxy using the tproxy mode. (tcp proxy is not an option) The downside of the tproxy side-car is that it needs the net\_admin capability in a pod - which is tolerable for standard use-cases, but in our case it is not, as a user will have direct access to a shell in that pod and therefore capabilities need to be highly restricted. **For this reason I was thinking whether it is possible to write the ip-tables rules ( right now done by the side-car tproxy) via a CNI-Plugin**, so that the pod itself would not need the net\_admin capability. From my current knowledge, this is also what service meshes like linkerd and istio are doing. > > > \## Feature Request > > \### What problem are you trying to solve? > > Many productiโ€ฆon clusters are locked down and do not allow arbitrary pods to have \`cap\_net\_admin\`. This makes it, unfortunately, impossible to use linkerd in these environments. > > \### How should the problem be solved? > > It would be nice to have an option (addon?) that does the iptables configuration outside of a pod's initContainer. This would allow an operator with elevated privileges to add the solution to the cluster and have arbitrary pods have minimal privileges. > > A good solution would be creating a CNI plugin that does the iptables setup. A DaemonSet would be added to the cluster which adds the CNI plugin to the set. Each time a pod is scheduled, the iptables rules would be added as part of network setup. This allows for them to be in place when the pod starts up (there might be some oddities with initContainers?). > > Take a look at istio/cni for some good previous work. > > \### Any alternatives you've considered? > > \- DaemonSet controller that updates iptables - Obviously the easiest solution, there are real sync problems and issues with pods starting that don't have rules in place (making policy a little difficult). > \- HTTP\_PROXY options - Would only work for outgoing traffic, which isn't great. > > \### How would users interact with this feature? > > This should be optional and not default. It would be an awesome use of addons: > > \`\`\`bash > linkerd add secure-cluster-helper > \`\`\` > > There are definitely some big open questions about the interactions between this and things like the auto-inject feature. BR Jan ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [August 25, 2023, 12:48pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/2 "2023-08-25T12:48:29Z")
Welcome back, @janst. The plugin idea looks promising, and I'll need to catch up on how those work to say more. Meanwhile, did you rule out the possibility of an opaque forward proxy port, or is that what you were referring to when you said TCP proxy isn't an option, e.g., `127.0.0.1:3000` where 3000/tcp is a TCP proxy port for a particular Ziti service provided by a sidecar running `ziti tunnel proxy`? I understand the zero-trust constraint applies to the pod level, so a cluster-level proxy is not an option. ---
**Author:** ![janst](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/janst/32/3076_2.png) [@janst](https://openziti.discourse.group/u/janst)\ **Post date:** [August 25, 2023, 1:04pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/3 "2023-08-25T13:04:31Z")
Hi Ken, yes I meant that `ziti tunnel proxy` is not an option, as our application would need to know somehow, which port to speak to... - at least I cannot image how that could work. Can you image any other solution we could go with, despite from the TCP proxy port ? (for the time being) Probably by integrating a service mesh like istio or cilium ? (that's anyway something we are thinking about) BR ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [August 25, 2023, 2:23pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/4 "2023-08-25T14:23:31Z")
I don't see an alternative if `NET_ADMIN` can't be granted to the pod. As I understand, that's needed to install the `TPROXY` rules. I'll poke a couple of colleagues in case they have alternatives to offer. It makes sense to me for us to adopt the pattern used by Istio because their proxy sidecar serves a similar purpose, only the Ziti sidecar is intentionally split-tunnel, not requiring all destinations to be on-mesh like Istio's sidecar, with external exceptions only. Rather, Ziti's sidecar allows any destination and only "intercepts" Ziti service addresses. ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [August 25, 2023, 2:38pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/5 "2023-08-25T14:38:23Z")
I see that a hypothetical Ziti CNI could be chained to whatever existing CNI, e.g., Calico, Flannel. That's good news because I had believed the cluster admin had to choose exactly one CNI or use a more complicated multiplexing CNI framework. ---
**Author:** ![janst](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/janst/32/3076_2.png) [@janst](https://openziti.discourse.group/u/janst)\ **Post date:** [August 29, 2023, 5:49am UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/6 "2023-08-29T05:49:53Z")
That sounds promising! So in case we have Cilium or Istio in place we could use the "ServiceMesh CNI plugin" and the hypothetical Ziti CNI plugin at the same time. Do you see a way of implementing such a Ziti CNI plugin for validating this expected behavior ? I think that such a feature could greatly improve the interoperability with K8s ๐Ÿ™‚ ---
**Author:** ![dariuszSki](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dariuszski/32/339_2.png) [@dariuszSki](https://openziti.discourse.group/u/dariuszSki)\ **Post date:** [August 29, 2023, 12:55pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/7 "2023-08-29T12:55:32Z")
Ziti is a service mesh and one needs ziti router with edge/tunnel to be deployed in a cluster to enable that mesh between microservices in that cluster at the very least (similar to the ambient mesh) . We also have the ebpf interception now (more about it here `https://github.com/netfoundry/zfw`) that is available at the VM level at the moment. We are thinking about how to use it in k8s natively. The cni plugin looks interesting. Also, we are trying to gauge what is the percentage of k8s users that prefer node level proxy vs pod level as well. ---
**Author:** ![janst](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/janst/32/3076_2.png) [@janst](https://openziti.discourse.group/u/janst)\ **Post date:** [August 29, 2023, 2:09pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/8 "2023-08-29T14:09:59Z")
Hi @dariuszSki, thanks for your response! Great to hear that you are thinking about these these things. I can definitely say, that I would prefer to have a pod-level proxy, otherwise there wouldn't be an end-to-end trust between services - which is actually a major feature of OpenZiti. Have you already tested the ebpf interception within K8s - meaning is there a repository available ? Are you planning on implementing a CNI plugin for Ziti to test whether that could work ? I'm asking, as there is currently no possibility for K8s pod egress traffic interception without granting additional capabilities to a pod - despite the LoopbackProxySidecar, which does not provide DNS. BR Jan ---
**Author:** ![dariuszSki](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dariuszski/32/339_2.png) [@dariuszSki](https://openziti.discourse.group/u/dariuszSki)\ **Post date:** [August 29, 2023, 3:45pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/9 "2023-08-29T15:45:50Z")
Hi Jan, Yes, we are thinking/brainstorming for sure and definitely encourage inputs/suggestions to see what make sense long term for egress side of things for most users. We have the ebpf code that can be deployed at the vm level. We have not started the work yet for containers. Most likely would be at the node level, since you need elevated permissions, i.e. CNI or CNI plugin. As you know yourself that CNIs like Cilium use ebpf too. We are not clear how it would work yet if we were to develop a plugin. You can run multiple ebpf programs in sequence today and there is also more work being done in that space to enhance that to mix and match ebpf programs. It may not be an issue at all. Here is the source code if you want to have a look or even contribute [zfw/src](https://github.com/netfoundry/zfw/tree/main/src). One thing we always ask for the pod level proxy users, why not embed ziti listener/dialer into apps/microservices? By doing it, all the networking connectivity issues kind of go away. ๐Ÿ˜‰ Thanks, Dariusz ---
**Author:** ![janst](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/janst/32/3076_2.png) [@janst](https://openziti.discourse.group/u/janst)\ **Post date:** [August 31, 2023, 11:46am UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/10 "2023-08-31T11:46:35Z")
Thanks for the reply and being open about the progress on this topic. I'm definitely willing to contribute, as this is sth. me and my colleagues are dependent on ๐Ÿ™‚ I will try to a push this and create a task for it in our Kanban - will come back to you as soon as I can work on it ๐Ÿ™‚ BR Jan ---
**Author:** ![dariuszSki](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dariuszski/32/339_2.png) [@dariuszSki](https://openziti.discourse.group/u/dariuszSki)\ **Post date:** [August 31, 2023, 3:48pm UTC](https://openziti.discourse.group/t/cni-plugin-for-openziti/1552/11 "2023-08-31T15:48:43Z")
FYI, this is a relevant project to this as well. [`https://bpfd.dev/`](https://bpfd.dev/)