Kubernetes cluster over OpenZiti Network

1

Hello.

I am trying to create a k8s cluster over an OpenZiti network:
Let's say that I have 1 node called A in one network and 1 node called Bin another network. Those two networks are not connected to each other. So A cannot access B by default.

Now, both nodes are connected to an OpenZiti overlay network, and I have set up a simple http service on A and I am able to access it from B.

Similarly, following the tutorials you have on your pages, I am able to instantiate a k8s cluster on A and use kubectl from B.

What I want is to join B as a worker node to the cluster of A. I wasn't able to find any information on that on your pages.

Using the same setup as the one for kubectl, I am able to actually join using a DNS name. So kubectl get nodes, returns the two nodes. But then when I install a cni, there are problems, as the pods do not initialize correctly and the cni fails...From my understanding these problems arise from the fact that only port 6443 is not enough for k8s to establish a cluster.

To no avail, I have tried creating a bi-lateral service for flannel based on the ports described here: https://github.com/flannel-io/flannel/blob/master/Documentation/troubleshooting.md#firewalls

Do note that I am not using a deployment of the tunneler/router on k8s or docker. Just regular host services.

Has anyone tried this? Do you have any tips as to what services are needed to be open for such a thing to work?

Also, to a similar extent, is there a way to define a bunch of services altogether? Or should I define each one by one? I found for example a lot of port that are needed to be open here:

Can I create a single service for those?

2

I did something like this with the Rancher installer CLI a couple of years ago. It worked, but most of my K8s experimentation with Ziti has used nodes homed in a shared subnet. If you're able to share more about the motivation for running remote nodes tethered with an overlay then I'd be glad to hear about it!

You'll need to define subnets for each group of nodes you want to connect with Ziti so that you can write Ziti service configs and policies accordingly. For example, nodes in subnet A should use the normal network to talk to their own subnet and use Ziti to talk to subnet B.

The two approaches you could take are

  1. running a Linux tunneller on each control and worker node, installed before joining the kubelet to the control plane, or
  2. run a Ziti L3 gateway for each subnet. If you take the subnet approach then the nodes in each subnet need static routes for the subnets provided by the Ziti gateway, unless it's the default L3 gateway.

My hunch is that the tunneller-on-each-node approach is best.

I believe it will work if you create broad Ziti services for the IPs and ports that will be used by the kubelets and kube-controller. Only the L4 transports used by K8s need to be specified as Ziti services, not the virtual networks that K8s will operate.

Here's an overview of such a broad Ziti service for subnet A. You need this for each subnet. It has these parts:

  1. A Ziti intercept configuration for clients, e.g. intercept if dest IP in subnet B
  2. A Ziti hosting configuration for servers, e.g. forward to dest IP and port
  3. A Ziti dial service policy granting permission to clients, e.g. allow subnet A to call subnet B
  4. A Ziti bind service policy granting permission to servers, e.g. allow servers in subnet B to host this service
  5. A Ziti service that associates the above together as an entity

The first two are the tricky parts because they use specific syntax. I'll give CLI examples of creating those. Let me know if it's unclear how to create the policies and the service. Those are less tricky.

I didn't run or test these, so it's possible there's a mistake in there. Hopefully, it gets you going!

The upshot of this config for subnet A is that any IP with destination in subnet B (10.11.12.0/23) should be intercepted by Ziti.

ziti edge create config "subnet-a-intercept-config" intercept.v1 '
{
        "protocols":["tcp","udp"],
        "addresses":["10.11.12.0/23"],
        "portRanges":[{"low":1, "high":65535}]
}'


ziti edge create config "subnet-a-host-config" host.v1 '
{
        "forwardAddress": true,
        "allowedAddresses": ["10.11.12.0/23"],
        "forwardPort": true,
        "allowedPorts": [{"low": 1,"high": 65535}],
        "forwardProtocol": true,
        "allowedProtocols": ["tcp","udp"]
}'

host config reference: The host.v1 Config Type | OpenZiti

Ziti running on Kubernetes quickstart: Kubernetes Quickstart | OpenZiti