RFI: openziti for dynamic inter-cluster communication

1

I've just learned about openziti and it looks very interesting. However, before I start digging deeper into it, I'd like to collect some information to assess whether it is suitable for the use case I have in mind.

Scenario: two kubernetes cluster, let's call them "master" and "edge". Regarding the physical network setup, "master" is on publically reachable IPs, so "edge" can open connections towards it (not the other way round).

My goal is to make some services in "edge" visible in "master", and occasionally the other way round too. The caveat is: the set of exported services can vary over time, so it should be possible to dynamically add and enroll new services to be exported and made visible from the other cluster (ie, without having to maintain a static list to update manually and/or redeploy or rerun CLI commands every time).

Is openziti suitable for this use case? If you need more details just ask.

2

Hi @mt2020, welcome to the community and to OpenZiti!

Yes. This is exactly the sort of thing that OpenZiti is suitable/perfect for. You would have services exposing the controller and you'd need one (or more) edge routers exposes through some kind of ingres on the 'master' cluster...

Then on the 'edge' cluster, you have numerous options as to what the best way to go about is for you. You could deploy a single router (or many routers) or ziti-edge-tunnels, or use pod-based deployments etc...

I think you'll find OpenZiti does exactly what you want/need.

3

Thanks @TheLumberjack . Can you point me to some docs on how to automatically enroll/expose new services without an operator having to create identities and run other ziti CLI commands? Ideally I'd like it to be done in a declarative way, as is typical for kubernetes, ie you annotate a service or edit some data in a ConfigMap (for example), and the magic happens (that is, the service appears in the other cluster). This is the critical bit for me, I can then study all the docs to make that happen.

4

I must admit, I sort glossed over this caveat:

rerun CLI commands every time

It's probably best to use an example. Can you give some examples of what sort of dynamic nature you're looking for?

For example, OpenZiti supports "wildcard" intercepts. So you can configure an intercept such as: "*.my.intercepted.svc" so that "database.my.intercepted.svc" is intercepted and sent to the edge cluster, but then if "httpserver.my.intercepted.svc" shows up, it'd also be tunneled.

If that's what you're after, then you just set it all up once and there's no configmap to update.

I don't think we have anything that would automate provisioning of ziti services from a configmap yet, but that sounds like a neat bit of tooling. it does put some trust in the kubernetes cluster on the whole, but I can understand why that'd be useful for people who want that sort of thing. @qrkourier has more experience with hands-on kubernetes than I do, he might have other ideas. We have a couple other community members using kubernetes for somewhat similar needs -- if they see this post they might offer other thoughts.

5

Yes, something like the wildcard intercepts sounds like it might do the trick (note: without having read the docs, which I'll start doing ASAP now).
The idea is that clusters will be orchestrated by a third-party tool (for example using GitOps or deploying helm charts, etc, but generally speaking "not using CLI"), so things should be done as declaratively as possible, so to speak.
I'll keep you posted. Thanks!

6

After some deep-dive, I've been able to set up a controller+router running on the "master" cluster, and a tunneler (the ziti-host helm chart) running on the "edge" cluster.
Now I'm trying to create the configs so that traffic can be forwarded from one cluster to the other. For the interceptors, I've created the following configs:

$ ziti edge create config tcp-intercept-e2m intercept.v1 '{"protocols":["tcp"],"addresses":["*.master.ziti"], "portRanges":[{"low":0, "high":65535}]}'
$ ziti edge create config udp-intercept-e2m intercept.v1 '{"protocols":["udp"],"addresses":["*.master.ziti"], "portRanges":[{"low":0, "high":65535}]}'
$ ziti edge create config tcp-intercept-m2e intercept.v1 '{"protocols":["tcp"],"addresses":["*.edge.ziti"], "portRanges":[{"low":0, "high":65535}]}'
$ ziti edge create config udp-intercept-m2e intercept.v1 '{"protocols":["udp"],"addresses":["*.edge.ziti"], "portRanges":[{"low":0, "high":65535}]}'

That should take care of picking up traffic destined to the "other" cluster and send it into the tunnel. However, I'm having trouble defining the host.v1 config, as I can't see a way to determine where the traffic coming from the tunnel should be offloaded, as it can be for a variety of services and different ports.

$ ziti edge create config tcp-e2m host.v1 '{"protocol":"tcp", "address":"??????", "port":???}'

Should I create a config for each possible target address and port? Or is there something I'm missing?

7

I expect you know, but you could condense that tcp/udp into only two services if you want:

As two services:

ziti edge create config tcp-intercept-e2m intercept.v1 '{"protocols":["tcp","udp"],"addresses":
[".master.ziti"], "portRanges":[{"low":0, "high":65535}]}'
ziti edge create config tcp-intercept-m2e intercept.v1 '{"protocols":["tcp","udp"],"addresses":
[".edge.ziti"], "portRanges":[{"low":0, "high":65535}]}'

The host.v1 Config Type | OpenZiti has an example of the host.v1 side. You will want to forward port, protocol and address. The "forward*" options basically indicate "whatever you intercept, i will offload on the far side". So in your example, if you intercept database.edge.ziti from master, that will be sent to edge and then offloaded towards database.edge.ziti on the far side. same would be true if you relied on IP addresses fwiw, if you intercept 10.10.10.10 on master, it'd be tunneled to edge where the edge tunneler would try to connect to 10.10.10.10... So whatever is intercepted, gets sent out.

Here is the example provided on that page:

{
  "forwardAddress": true,
  "allowedAddresses": [
    "192.168.1.0/24",
    "10.0.0.1/16"
  ],
  "forwardPort": true,
  "allowedPorts": [
    {
      "low": "1024",
      "high": 2048
    }
  ],
  "forwardProtocol": true,
  "allowedProtocols": [
    "tcp",
    "udp"
  ]
}
8

Ah great, I wasn't aware of the forward* options. However, I seem to understand that with this setup, to minimize the amount of manual configuration, I should then use the same actual kubernetes names on the interceptor side, that is, intercepting *.somenamespace.svc would send traffic to (existing) services living in the same namespace and with the same name on the other cluster, because if I use custom names and intercept eg *.master.ziti at the edge, I would then need to (manually) create corresponding blah.master.ziti etc on the other side (and connect them to the right pods).
So I should create interceptors for all namespaces that host target services on the other cluster, which could be any. Besides this, what happens if one of those namespaces also exists locally? How can I access local services in that namespace without being forwarded into the tunnel?

9

That's what I'd probably do. That seems like it solves your other problem too of "what happens when i want local services not remote services"? I'm no kubernetes namespacing/DNS wizard, and I don't have a full picture of what you're actually doing so it's somewhat hard for me to have a good enough mental model to answer that really precisely.

If they both exist, the intercept will take precedence. Often, people want that behavior. I don't know if I've seen a setup where people didn't actually want that setup... I think in that case they just don't intercept that particular address... Do you have a use case for why you'd want to have that sort of control where "some services are tunneled and some aren't"? I need a solid usecase to be able to come up with something for that one. I'm struggling to understand the 'why' behind that. I'm certain you have a good reason, I just don't see it yet :slight_smile:, or maybe I'm just blocked on the example for whatever reason. Can you explain the why/how you want it to work with an example?

10

Sure. Well, to be honest I'm just trying to understand how things work and the implications, to see if that would be compatible with the real use cases.

To use a concrete example, if the interceptor takes precedence and I were to intercept and forward *.default.svc (or even just kubernetes.default.svc, for that matter), then all of a sudden pods would lose the ability to connect to their local kubernetes API service (being able to see the other cluster's kubernetes API service is actually a real use case).

That's why it would be beneficial to have some sort of mapping/indirection between names as they appear on one side and the real names on the other side.

Generally speaking, the use case is being able to access clusterIP services on the other cluster without the need to expose them via ingresses, NodePort or LoadBalancer (think collecting prometheus metrics, access the kubernetes API server, access dashboards or other internal HTTP services). Why? Because sometimes you don't have the needed admin rights to make changes on the edge, or even if you do, the edge might not be externally reachable (only outbound connectvity), or you don't want to go through those steps for a variety of reasons (eg they might only be needed for a limited time).

So the idea is that you own the "master" cluster (full control), and you want to manage (or "do something with") edge clusters from the master. As said, you might not own the edge cluster, and you might not have privileged access to them (eg you might only access a single or a few namespaces), because some customer manages it, or you might also have privileged access; all cases might happen.
In any case, you install some unprivileged "agent" (which doesn't require elevated or cluster-wide permission, so no L3 networking, no CRDs, etc) on the edge that could give you a protected entry point into the cluster; once you have that, depending on the access level, you might want to do things directly against its k8s API (eg deploy applications/helm charts), or "simply" access remote services, depending on the actual scenario. You might also need to make some master services visible on the edge. In all cases, it's always a matter of being able to see and access (kubernetes) services that exist on the other side. I know this all sounds vague, but it's because it's meant to be a very generic mechanism for more complicated logic to be built on top of it.

One tool that seems to offer similar characteristics and capabilities is skupper (https://skupper.io), which I'm also evaluating, although that is not a zero-trust solution like openziti.

Thanks to your suggestions I can proceed further and configure the host.v1 services. I'll let you know how it works out. Thanks!

11

Hmm. Yah, I'm not 100% kubernetes-fluent but I can understand where you're coming from. Thx for the follow-up. If you don't mind, can you outline what sort of config-map updater you have in mind? It might be a relatively simple thing for me to create. My initial reaction is something that simply automates the ziti cli or uses the golang SDK to do the work (which would be another way to do it) might be easy enough. I'm not a k8s n00b but i'm also not a long-time k8s admin... but if it's easy enough to do, i might be able to demonstrate how that works and it sounds interesting and fun... It'd only be an 'alpha' type thing for now though. So as long as you understand that - it might be worthwhile if it's easy enough, I might be able to take it on.

sounds good.

12

FWIW I looked at Skupper in the past, which provides a layer-7 service multi-cluster interconnection service for secure communication across Kubernetes clusters by defining an ad-hoc virtual networking substrate just for a specific set of namespaces (i.e., it does not introduce a cluster-wide interconnection). Skupper implements multi-cluster services in namespaces exposed in the Skupper network. When a service is exposed, Skupper creates particular endpoints, making them available on the entire cluster. TL:DR, Skupper provides L7 DNS-style overlay which decouples the application layer network from the underlying network infrastructure to allow secure comms without a VPN "

As you say, it's not focused on zero trust nor can it operate L3/4 as Ziti does. There are probably many other differences but its been a while since I looked into it.

13

Hi @PhilipGriffiths yes, that's pretty much it, even today. The convenience of skupper is that a single trusted tunnel is established once, then everything is dynamic; eg, you annotate a service in one cluster, skupper picks that up, transmits it to the other cluster where it immediately appears with a name of your choice (specified in the annotations) and is ready for use. You connect to it, and you are transparently forwarded to the (real) annotated service at the other end.
(skupper has other aspects that I don't like too much, such as the fact that it forms a full mesh without too much control over which service is propagated to which cluster, so you might end up seeing unrelated services in places where you're not supposed to see them)

@TheLumberjack this would be something open to discussion; the controller could receive a special request or watch for particular API objects to appear or be updated (or annotations) and act accordingly; since this is something that is not present today, I don't have any preferred method to do that, other than kubernetes-native (ie YAML-based) would probably be better.

14

Something that occurs to me right now which might be simple and low-impact to implement, and could go a long way, could be to add some string mangling mechanism (eg regex-based) upon exit, to remap the requested names. Something like (just making up the syntax here, but you get the idea):

{
  "forwardAddress": true,
  "forwardPort": true,
  "forwardProtocol": true,
  "transform": "s/\.my-edge-cluster//"
}

so I could intercept eg *.my-edge-cluster.svc on the master (eg whatever.some-edge-ns.my-edge-cluster.svc), and have it reach whatever.some-edge-ns.svc when it gets out of the tunnel (or any other transformation, as long as the original string contains enough information to reconstruct the target service name). Port remapping might also be useful.
If you think you might be interested in implementing that or something along those lines, I'll gladly volunteer to test it.

15

It is the kinda thing that isn't a priority for me right now, but it sounds like fun so it might be something i do on the 'nights and weekends' sort of thing. If I get to it -- or should someone else get around to implementing something like this, I'll definitely let you know! :slight_smile:

1 Like