I'm working on a project that will utilize openZiti in Kubernetes environment but I'm not certain about setting up 1 edge service that will call other services via rest inside kubernetes.
Should I design the microservice "Security Service" and "Communciation Service" to register as tunneller and remove the Edge Service (the one with go executable) ?
I still have a requirement with c# application to use pulsar, but directly calling it outside kubernetes, may break zero trust principle.
This is a classic "ZTNA" type of setup. Is "Edge Service" an edge router or is it a ziti-edge-tunnel or are you going to make your own go-executable to do that proxying? I ask, because if it were me, I'd just deploy a "private" edge router in that scenario. You can use the edge router to offload traffic (proxy) inside kubernetes if you want to go the "ZTNA" route. That's a very common pattern people start with and should be very quick and easy for someone with your understanding of OpenZiti to setup and start prototyping.
I personally find that I need to actually set it up and try it before I either find subtle flaws in my original thinking that make me want to restructure a solution or that make me think that it was the correct path.
I don't know if we can advise you on how to design your services very well as we're not going to be experts in your overall design.
hope that helps
Thanks, I updated my design to use docker image: openziti/ziti-edge-tunnel. A pod can be designed to have multiple container setup, so the 1st container can be a web server and the 2nd container for ziti tunneller, the setup allow the pod to select which container to use based on rule.
I managed to run this setup, but ideally, the image can receive additional argument that if the json already created then it will skip the enrollment, and simply load the existing certificate possible @TheLumberjack ?
Note that the volume mount is persisted; so even after restart the certificate can be used again so ideal process should simply ignore the jwt file if the json exist.
Here's a sidecar example that uses the IPtables capability of ziti tunnel to isolate the interception at the pod level with pod-local Ziti DNS.
That example shows adding Ziti to the pod deployment manifest similar to your example, only it uses the openziti/ziti-tunnel container image.
ziti-edge-tunnel in KubernetesThe openziti/ziti-edge-tunnel image you used in your example is the one that I would use for a node proxy because it uses IP routes to a tun device, which requires elevated privileges and affects all pods on the node. There's a way to accomplish a node proxy, rather than a pod sidecar proxy, with this Helm chart that creates a privileged daemonset in conjunction with K8s NodeLocal DNSCache to ensure that the Ziti nameserver on each node resolves Ziti DNS queries. This is important because Ziti intercept IPs are unique to the node only, not globally across nodes.
Thank you for sharing links will sure check. I'm just curious if there's a way to modify the behavior on openziti-tunnel image that if the json token already exist, then it will ignore the jwt enrollment. There might be some cases where the stateful container will restart and need to ensure that it wont fail during boot by checking the existence of json file.
Yes. That's how both images work. 
The pod sidecar proxy uses openziti/ziti-tunnel and the example guides you to store an enrolled identity as a k8s secret for the sidecar to mount.
The node proxy uses openziti/ziti-edge-tunnel and accepts either an enrollment token as input value or the name of an enrolled identity stored as a k8s secret.
