Securing Kubernetes with OpenZiti

[arslane]

Hey,

I’ve a Kubernetes Cluster with three master nodes, 10 worker nodes and 3 etcd inside the master node VPSs.

My Master Nodes are located in South of France, same for 5 of my worker nodes. 5 left worker nodes resides in Germany.

My goal while learning how Ziti works is to provide a security layer for my Kubernetes cluster.

First, I came accross this video Secure Kubernetes Cluster using Ziti, so this, can be used to make kubectl DARK, but is it enough ?

I mean isn’t it better to provide a Zero Trust Host/Network access for the whole cluster ? I thought about the architecture above.

• Edge Controller running in France (Idk yet about exact location)
• Private routers running on every single Node
• Fabric Routers in Paris (North of France) and Strasbourg (North East, very close to germany)
• One Edge Router in France (Idk yet about exact location)

My Private routers should have tunnel enabled, link.dialers aswell as listeners.binding = edge.

As you can see every node of my cluster has a Private Router, knowing that private routers can only dial other routers, I’m not sure that my Master nodes would be able to reach my workers. (On the docker compose example, we cannot reach blue network from red). Should I set link.listeners instead of link.dialers ? Also, what happens If i unset the listeners.edge = edge and keep the tunneling enabled ?

I mean, I don’t really want to have edge routers available for the outside world in my Cluster nodes VPS.

Feel free to edit the diagram using in diagrams.net and share it back.

Regards,
Arslane

[TheLumberjack]

As long as you have one or more public routers, your master nodes should be able to reach any private nodes. Reach shouldn't be any issue. In the docker example you could definitely reach the blue network from red but only by using ziti. We don't have any guide for doing something like that yet but I could try to show you how it you want. You just can't access the blue network directly from the red.

You should definitely not set link listeners for the private routers. You'll just end up with errors in your logs because the other routers won't be able to form links to the private routers.

I don’t really want to have edge routers available for the outside world in my Cluster nodes VPS.

This is one of those statements that needs some nuance. You should have no fear providing public routers to the Internet. The strong identity, combined with the need for authorization of services effectively blocks your routers from the outside world. But, an identity with the proper authorization would be allowed. So I just want to make sure you understand what I mean there?

I mean isn’t it better to provide a Zero Trust Host/Network access for the whole cluster ?

It ABSOLUTELY is better. You're already nearly there! With that video there's no way to access the cluster, or the work loads within without using OpenZiti. The Oracle kubernetes manager allows you to have a kubernetes cluster with the API entirely dark. Only though ziti can you hit the api.

Hopefully I caught all your questions, but if you want additional detail, I'll be happy to provide it.

[arslane]

@TheLumberjack

We don’t have any guide for doing something like that yet but I could try to show you how it you want. You just can’t access the blue network directly from the red.

If you can show me that it'd would be extremely helpful

@TheLumberjack

This is one of those statements that needs some nuance. You should have no fear providing public routers to the Internet. The strong identity, combined with the need for authorization of services effectively blocks your routers from the outside world. But, an identity with the proper authorization would be allowed. So I just want to make sure you understand what I mean there?

Yeah, I totally agree with you, but I feel that the private routers provides more advanced security features, as no one can directly connect to them.
So you're telling me that It's perfectly fine If I just use Public edge routers instead of private routers ?

[gnz]

Clint will get back to you on those aspects. Clarifying question: is your intent to secure both your management plane and the workloads? If so, this article (with embedded video snippets) of how IBM uses OpenZiti for multicloud zero trust Kubernetes may be of value.

[arslane]

I believe that this video show how to expose a zitified application living in kubernetes through Ziti, it’s pretty neat, but that’s not exactly what I want to do.

I mean, I don’t want to bother my self with tunnel sidecars or namespace specific tunnel pods. I feel like I’d better be having Ziti managin all traffic that come to/from my K8S cluster, so I want to secure my hosts and by the same occasion K8S management plane and workloads.

If you thing that I could be missing up something, I’d be glad to hear why I should use those instead of setting up a ZTHA

[TheLumberjack]

@arslane

I feel that the private routers provides more advanced security features, as no one can directly connect to them.
So you’re telling me that It’s perfectly fine If I just use Public edge-routers instead of private routers ?

I feel like we might have a disconnect. My point was that you will need to have at least one public edge-router (as your diagram shows), which all the "private" routers connect/link to. There are only two things in my view that makes a router "private" or a router "public". Note that I'm using quotes around these terms because these are not hard and fast terms.

A "public" edge-router will have a link listener and an edge listener and both will be addressable on the open internet. This allows "private" edge-routers to connect to these "public" routers, and it will allow edge clients to connect to these public edge-routers too.

A "private" edge-router can still accept edge connections from edge clients and it could accept links from routers if it wanted to, but probably not because a private edge-router exists in private network space without inbound holes in the network's firewall. Only devices on the same network would be able to connect.

So I wouldn't configure those edge-routers as 'public' because I would not provision any ingress controllers in your kubernetes cluster, keeping it totally dark.

Red NW to Blue

As for accessing the blue network, I was just saying that you can put a client into the red network, attach it to the OpenZiti network and then that red-client would be able to access the web-test-blue http server. On top of that, anything on the OpenZiti network which was authorized to reach the blue network would be able to communitcate to the things in the blue network.

The video Robert made also can show you how can use use an OpenZiti router as a LAN Gateway https://youtu.be/H0qGRBMGNIA

[arslane]

@TheLumberjack

I was just saying that you can put a client into the red network

Ohh okayy and Is it possible to intercept all traffic in ziti-private-red and route it to the correct endpoint (e.g ziti-private-blue) without edge client (since we're inside the container of the red private router) ? I mean, can the routers act like tunnelers, like intercept traffic going on the machine where it's installed and routing it to the correct endpoint ?

I already tried that with a service, without success, but I might be doing something wrong.

[TheLumberjack]

I don’t think you can do it in docker - or - if you can I don’t know how. Robert shows how to do it with “a real network” and by hand-crafting/updating the network as shown in that video above but I don’t know how to do that in docker (at this time).

I feel like it should be doable, but I’ve not invested enough time trying it myself.

[arslane]

Followed Robert’s video and it worked like a charm on the Cloud, gotta automate all of these with ansible now haha.

Do you think it’s worth having routers in my endpoints (my cluster nodes) or should i stick with edge tunnel clients ?

[TheLumberjack]

Personally, I think it’s ever so slightly easier to use ziti-edge-tunnel in my opinion. You don’t need a config file at all. Enrolling the jwt and starting it is all that’s needed. So if you have edge routers deployable and running - it’s probably just fine to keep 'em. At that point I don’t think there’s enough of a difference in either direction.

[markamind]

@TheLumberjack

The video Robert made also can show you how can use use an OpenZiti router as a LAN Gateway Using an OpenZiti router as a LAN Gateway - YouTube

After watching this video again.. I now understand what to do..

One difference that I need to address is that I will be running the local router on a MacOS computer.

Questions I have are

1. how to determine my local IP address

I think this is the correct command.. and the expected IP address.. does this make sense?

ipconfig getifaddr en0
192.168.20.2

2. how to configure the yaml for the local router

I have marked up the changes below using the local IP address above.

One area I am unsure about is the IP address 100.64.0.1 used for dnsSvcIpRange. I am assuming this is correct

The other bit that I am more uncertain about is lanIf: ens33. When I run ifconfig, I do not get any interfaces listed that have that name. However, I did find en0 matched to the 192.168.20.2.. so I have a hunch that this is what it should be

config.yml
  v: 3

  identity:
     cert: "certs/identity.cert.pem"
     server_cert: "certs/internal.chain.cert.pem"
     key: "certs/internal.key.pem"
     ca: "certs/intermediate-chain.pem"
  edge:
    csr:
      country: US
      locality: Charlotte
      organization: Netfoundry
      organizationalUnit: ADV-DEV
      #province: NC
      sans:
        dns:
          - "localhost"
        ip:
          - "127.0.0.1"
          - "192.168.20.2"
  ctrl:
    endpoint: tls:<controller-ip>:80

  link:
    dialers:
      - binding: transport

  listeners:
    - binding: edge
      address: tls:0.0.0.0:443
      options:
        advertise: 192.168.20.2:443
        maxQueuedConnects:      50
        maxOutstandingConnects: 100
        connectTimeoutMs:       3000
    - binding: tunnel
      options:
        svcPollRate: 15s
        resolver: udp://192.168.20.2:53
        dnsSvcIpRange: 100.64.0.1/10
        lanIf: ens33

[markamind]

@arslane

Followed Robert’s video and it worked like a charm on the Cloud, gotta automate all of these with ansible now haha.

At the time I created this post, I was having problems running the remote router.. but I now have this working.

So.. no need to respond

Configuring LAN Gateway on Oracle Linux

I am working through the instructions in the video produced by @rcsoleng. After setting up the remote router, I receive the following error when I run it. FATAL edge/tunnel/dns.NewDnsServer: dns server failed to start: listen udp 10.0.0.121:53: bind: permission denied What have I done to attempt to resolve this Checked that port 53 on UDP is open on the firewall and VCN (checked) Other tips? I really feel like I am walking in the dark here… I found this reference in the Oracle help docu…