I have four Kubernetes clusters, each with a Ziti router, all connected to the Ziti controller in the main cluster. I have successfully configured services and policies for all the Kubernetes workloads we want users to access.
We want to allow SSH access to the nodes used in the cluster. Do you recommend I use the in-cluster routers, or should I install Ziti routers directly on the nodes? Having the routers run on the nodes has the benefit that I can still access the node via SSH if the cluster goes sideways, which makes me lean towards this. I'm curious if you would recommend the same configuration?
Everyone is different here. Sure you could use a router on all your nodes if you like but routers all participate in the overlay itself so you need to make sure you configure them accordingly. For that reason, I usually recommend using ziti-edge-tunnel just because it's simpler (imo) to deploy. You don't need to worry about which routers are linking to which routers, the controller won't have multiple paths to consider/calculate etc., just makes me think I'd use ziti-edge-tunnel.
There's also a kubernetes integration you might find useful that injects sidecars through a webhook. It still in early stages but it should do that sidecar-injection that you might be interested in having a look at. I've not played with it myself, but @qrkourier is quite familar with it and can defintely add more value on that than I can at this time. 
I'll point this thread to him and we can see what he thinks too.
I considered using ziti-edge-tunnel but wasn't sure if it would work since I've only used it as a client. I'll definitely try this first, because like you say, it is very easy to set it up and we only need it to expose SSH.
I had a look at the video @qrkourier made to demo the ziti k8s agent with sidecar injection and was impressed with what I saw. I hope I can find time to try it out soon.
You could include ziti-edge-tunnel.service, ziti-router.service, or both systemd units from our Linux package repo in your K8s node OS.
There's some overlap in their functions, e.g., either can host or provide access to services at the host level, and I agree the tunnel service is probably best if your main goal is to provide remote access via an SSH service.
For my nodes, I like to create a "{name}-node-local" service that re-uses the same host.v1 config for each node-specific service:
{
"address": "127.0.0.1",
"allowedPortRanges": [
{
"high": 65535,
"low": 1
}
],
"allowedProtocols": [
"udp",
"tcp"
],
"forwardPort": true,
"forwardProtocol": true
}
If I have only a few nodes, I may craft a separate such remote-access service for each node. If I have many nodes, then I'll use one service for all and the 'addressable terminator' feature of Ziti to connect to a specific node with that service that's common to all nodes.
BTW, I've finally released the Helm chart to install ziti-webhook (a sidecar injector that manages Ziti identities on behalf of pods).
Doc: GitHub - netfoundry/ziti-k8s-agent
e.g., to install the sidecar injection webhook:
# Create secret from enrolled identity JSON file
kubectl create secret generic netfoundry-admin-identity \
--from-file=netfoundry-admin.json=netfoundry-admin.json \
--namespace=netfoundry-system
# Install webhook from latest release
helm upgrade --install --namespace="netfoundry-system" --create-namespace ziti-webhook \
https://github.com/netfoundry/ziti-k8s-agent/releases/latest/download/ziti-webhook-chart.tgz \
--set identity.existingSecret.name="netfoundry-admin-identity"
1 Like
Thank you for the helpful feedback @qrkourier!
I realise I'm not understanding something. If the address is 127.0.0.1, like in your example, how does the controller know where to route traffic for the service? By comparison, I use argocd-server.argocd.svc.cluster.local as the address for my host.v1 config, which points directly to the service in the cluster. If I deploy ziti-edge-tunnel on each node, I assumed I would have to create a host.v1 config for each node and specify the node address (eg, 172.16.0.1) in each config. Or is this magic possible through the 'addressable terminator' feature?
I'm looking forward to trying out the webhook!
You're asking about addressable terminators, which is a way for a single service to provide discrete access to individual hosting identities. Without this feature, the service is normally load balanced across all the hosting identities. With addressable terminators, you can target a specific hosting identity, so it's a way to provide remote access to all your nodes with one service.
To use addressable terminators:
-
In the service's intercept.v1, set:
"dialOptions":{"identity":"$dst_hostname"}}'
This means the service's intercept address is hosting identity's name.
-
In host.v1, set:
"listenOptions": {"bindUsingEdgeIdentity":true}}
-
Grant bind permission to an identity. Give the hosting identity a name like a domain name, e.g., node1.ziti.internal, so it will be unambiguous in Ziti DNS when dialed.
-
Then, dial the specific binding identity by name in Ziti DNS, ssh node1.ziti.internal.
There's an interesting experiment that demonstrates a slight variation on the above strategy. zssh might be helpful since you are aiming for SSH via Ziti. zssh doesn't rely on the intercept.v1 config to provide DNS on the client side, but instead dials the hosting/binding identity by name with the Ziti SDK (https://github.com/openziti-test-kitchen/zssh?tab=readme-ov-file#readme).
Amazing! Thanks for the detailed instructions!
I haven't looked at zssh because I don't want to burden users with installing additional clients.
@qrkourier @TheLumberjack This is fantastic! I'm successfully exposing SSH using addressable terminators!
I want to revisit my original question to make sure I'm not deploying unnecessary components.
My setup looks like this now:
Staging, Storage1 and Storage2 k8s clusters
- all nodes run ziti tunneler directly on host as systemd service
- router runs in k8s cluster, installed with helm chart
Production k8s cluster
- all nodes run ziti tunneler directly on host as systemd service
- router runs in k8s clsuter, installed with helm chart
- controller runs in k8s clsuter, installed with helm chart
k8s services are configured to target specific routers:
node-level services (like ssh) use addressable terminators, and there is a bind and dial policy for each node:
They share a node-local-host host.v1 config and have an intercept per node:
Is this how you would deploy it? Or can you suggest some optimisations?
You do have some unnecessary service policies and intercept configs.
You can use a single dial service policy for all dialers and bind service policy for all binders/hosters. Simply grant each with an identity role like #worker-nodes.
You're correctly using a single host.v1 config for all binders, targeting the loopback address and setting:
"listenOptions": {"bindUsingEdgeIdentity":true}}
You can use a single intercept config for all dialers setting:
"dialOptions":{"identity":"$dst_hostname"}}'
That way, no hard-coded intercept "address" is configured in the intercept.v1 config, and when a worker node has an identity named like "node1.ziti.internal", any authorized client can dial that domain name in Ziti DNS to reach its addressable terminator.
1 Like
This is nice option, im trying it, but when i add intercept , address field looks mandatory and its not allowing me to make it empty. btw Im using 1.15 controller version .
I suggest trying a dummy/placeholder address value if that version of the console requires you to fill in the address field. Hopefully, the $dst_hostname intercept will also work.
FWIW I'm not able to get it working with a single intercept with a dummy hostname and $dst_hostname in dialOptions either. I'm able to add the intercept but DNS is not updated for nodes with the worker-nodes attribute. It's not a priority for me, I just thought I'd mentioned it.
1 Like
I left out an important piece, and just verified this works in my environment.
In the intercept.v1 config. you must have two things for addressable terminators:
- At least one of the
addresses must match the hosting identity's name. I like to use a wildcard like *.ziti.nodes.internal which matches a hosting identity name like host1.ziti.nodes.internal.
- Set
"dialOptions":{"identity":"$dst_hostname"} to elect the identity name as the intercept matcher.
Here's a complete example.
host.v1:
{
"address": "127.0.0.1",
"allowedPortRanges": [
{
"high": 65535,
"low": 1
}
],
"allowedProtocols": [
"tcp",
"udp"
],
"forwardPort": true,
"forwardProtocol": true,
"listenOptions": {
"bindUsingEdgeIdentity": true
}
}
intercept.v1
{
"addresses": [
"*.ziti.nodes.internal"
],
"dialOptions": {
"identity": "$dst_hostname"
},
"portRanges": [
{
"high": 65535,
"low": 1
}
],
"protocols": [
"tcp",
"udp"
]
}
