# What would openziti arch for this look like

**URL:** <https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616>\
**Category:** Ziti Overlay\
**Created:** [May 30, 2024, 4:00am UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616 "2024-05-30T04:00:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 4:00am UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/1 "2024-05-30T04:00:45Z")

</div>

xyz is an application used for providing featured operations on client data whose infra is  
managed via k8s clusters and those every piece of data is made sure to be secured

Every client has a dashboard he can create multiple clusters across tenants and add plugings to drive  
usecase on his data

should we consider to install different controller to every client than keeping it centralized since  
if by chance some jwt gets exposed in one of the client clusters there might create attack surface

Or we should go with multiple controllers for each cluster approach , can one tunnel manage multiple controllers ?  
cant be used to connect to multiple clusters

How would the overall architecture look like ?

It would be really great if you can draw and send it how we should assemble the components

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [May 30, 2024, 11:30am UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/2 "2024-05-30T11:30:46Z")

</div>

> [@sadath-12](#):
>
> It would be really great if you can draw and send it how we should assemble the components

I don't think we would make direct recommendations like this.

> should we consider to install different controller to every client than keeping it centralized since  
> if by chance some jwt gets exposed in one of the client clusters there might create attack surface

I wouldn't. I always recommend having a single overlay network and using policies to segment the network. OpenZiti very nicely supports this sort of deployment/multi-tenant model imo. A rogue JWTs doesn't expose the cluster at all and isn't a worry. Those are individual identities and they can't do anything without authorization.

> [@sadath-12](#):
>
> Or we should go with multiple controllers for each cluster approach

Same answer. I would just have a single overlay network, a single controller that all the clusters can access. you _can_ do it this way, but it seems like it'd be a burden to maintain. Hard to know without understanding all the details.

> [@sadath-12](#):
>
> can one tunnel manage multiple controllers

I think you're asking if one tunnel can have identities from multiple overlay networks -- yes. All the tunnelers can have numerous identities from numerous overlay networks without too much of a problem but **BE CAREFUL**... If all the networks have the **same** intercept, then you'll definitely have problems. All the intercepts would need to be unique... That's the only thing to worry about in my opinion...

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 4:32pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/3 "2024-05-30T16:32:17Z")

</div>

thank you @TheLumberjack . Should I configure the tunnel in sdk in my application code so that it can talk to other cluster ?

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [May 30, 2024, 4:45pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/4 "2024-05-30T16:45:55Z")

</div>

I think you're asking about using an OpenZiti SDK and embedding it into your code? If so, **YES!!!** That's undoubtedly the **best** approach. That makes your code incredibly flexible. You can run it from anywhere, dev machine, docker, kubernetes, etc. It won't matter where or how you deploy your application then. That's a huuuge benefit of using an OpenZiti SDK. 🙂

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 4:52pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/5 "2024-05-30T16:52:05Z")

</div>

thanks @TheLumberjack .I'll look over that for sure .

I have an issue here I followed this guide [Deploy OpenZiti in Kubernetes with Ease Using k3d](https://blog.openziti.io/deploy-openziti-in-kubernetes-with-ease-using-k3d) . But the difference is I installed controller on AWS VM NODE\_IP is set to public ip of that machine .

Then the router I installed was in eks which is allowing the public traffic , here is the command  
helm upgrade --install "ziti-router" openziti/ziti-router   
--namespace ziti   
--set-file enrollmentJwt=./router.jwt   
--set edge.advertisedHost=router.edge.ziti.eks\_node\_ip.sslip.io   
--set edge.advertisedPort=3022   
--set edge.service.type=LoadBalancer   
--set linkListeners.transport.advertisedHost=router.link.ziti.eks\_node\_ip.sslip.io   
--set linkListeners.transport.advertisedPort=10080   
--set linkListeners.transport.service.type=LoadBalancer   
--set ctrl.endpoint="${Controller\_NODE\_IP}:6262"   
--set nodeSelector."kubernetes.io/hostname"="eks\_node\_hostname"

The error from the router side is  
[0.421] ERROR ziti/router/env.(\*networkControllers).connectToControllerWithBackoff.func2: {endpoint=[tls:controller\_ip:6262] error=[error connecting ctrl (tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not controller\_ip)]} unable to connect controller  
[92.826] ERROR transport/v2/tls.(\*sharedListener).processConn [tls:0.0.0.0:10080]: {remote=[eks\_node\_ip:54051] error=[EOF]} handshake failed

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [May 30, 2024, 5:04pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/6 "2024-05-30T17:04:26Z")

</div>

The value of the router's `ctrl.endpoint` must exactly match the controller's `ctrlPlane.advertisedHost`.

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 5:26pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/8 "2024-05-30T17:26:40Z")

</div>

thanks , I thought i did that . But when I retried I got it

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 5:27pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/9 "2024-05-30T17:27:43Z")

</div>

Next going to install router in private k8s cluster . Hope it works

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 5:33pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/10 "2024-05-30T17:33:34Z")

</div>

Ideally normal installation of routers like showed in that guide should be enough ? I saw some diagrams of public/private routers so confused about that

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [May 30, 2024, 5:50pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/11 "2024-05-30T17:50:21Z")

</div>

> [@sadath-12](#):
>
> thanks , I thought i did that . But when I retried I got it

Great! Glad to hear that. 🙂

The controller chart has two similarly named input values. The one that needs to match the router input is `ctrlPlane`.

- `clientApi.advertisedHost` - the DNS name of the controller's client API that must be reachable by all Ziti identities and Ziti routers
- `ctrlPlane.advertisedHost` - the DNS name of the controller's control plane listener that must be reachable by all Ziti routers

Security note: While it's possible to use magic wildcard DNS from [sslip.io](http://sslip.io) with public IP addresses in this way, using a DNS zone you control is essential for production.

> [@sadath-12](#):
>
> Ideally normal installation of routers like showed in that guide should be enough ? I saw some diagrams of public/private routers so confused about that

You can configure a router as public or private. It's not a precise way of describing the router configuration, but it usually means the router has a private or public IP. If you are hosting the router on a public IP, it's generally because you want it to provide a public link listener for the other routers to call.

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 6:00pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/12 "2024-05-30T18:00:08Z")

</div>

cool , thanks @qrkourier

So how would we deploy routers in private eks , to talk to controller at public in this scenario

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [May 30, 2024, 6:24pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/13 "2024-05-30T18:24:46Z")

</div>

Here's an example of deploying a router in a cluster where the router does not publish services outside the cluster; it only provides an in-cluster edge listener.

ℹ Configuring a router with a link listener is unusual if it's not reachable on a public IP.

```bash
helm upgrade --install "private-router123" openziti/ziti-router \
--namespace ziti \
--set-file enrollmentJwt=./router1.jwt \
--set edge.advertisedHost=private-router123-edge.ziti.svc.cluster.local \
--set linkListeners.transport.service.enabled=false \
--set ctrl.endpoint="{{ ctrlPlane.advertisedHost }}:6262"

```

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [May 30, 2024, 7:49pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/14 "2024-05-30T19:49:01Z")

</div>

> [@qrkourier](#):
>
> ```auto
> helm upgrade --install "private-router123" openziti/ziti-router \
> --namespace ziti \
> --set-file enrollmentJwt=./router1.jwt \
> --set edge.advertisedHost=private-router123-edge.ziti.svc.cluster.local \
> --set linkListeners.transport.service.enabled=false \
> --set ctrl.endpoint="{{ ctrlPlane.advertisedHost }}:6262"
> 
> ```

worked , thanks alot . Can I interact with private cluster's apiserver and do kubectl commands from other vpc using openziti ? Maybe exposing apiserver url not sure how it would work

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [May 30, 2024, 7:51pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/15 "2024-05-30T19:51:46Z")

</div>

Yep. The Kube API server is published by default as ClusterIP service `https://kubernetes.default.svc.cluster.local`. You can create a Ziti service that targets `kubernetes.default.svc.cluster.local` on port `443` and it becomes available to the devices with dial permission from a Ziti service policy.

Relevant resource: `kubeztl` is a fork of `kubectl` with the Ziti library so it doesn't need a proxy to use your Ziti service for the Kube API: [GitHub - openziti-test-kitchen/kubeztl: A zitified kubernetes client](https://github.com/openziti-test-kitchen/kubeztl/?tab=readme-ov-file#readme)

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [June 1, 2024, 5:46am UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/16 "2024-06-01T05:46:20Z")

</div>

@qrkourier thanks ,

Let's say 2 services under 2 different private clusters want's to talk to each other ,  
How would that be achieved and implemented

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [June 1, 2024, 2:34pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/17 "2024-06-01T14:34:47Z")

</div>

I think you want to access the Kube API of multiple clusters from inside each cluster. Although each cluster's Kube API has a predictable, standard DNS name `kubernetes.default.svc`, I would create a new DNS name for each Kube API to use with Ziti to avoid confusion.

You must follow your Kubernetes distribution's advice for adding a DNS name to the Kube API server certificate. For example, with k3s, this is done by adding a `--tls-san cluster1.ziti.example.com` option for the new DNS name.

With these new DNS names (subject alternative names [SAN]) added to each cluster's API server cert, you can create a Ziti service for each name.

Assuming you have two clusters, you must configure a reverse proxy for each Kube API and a proxy for each client pod.

I recommend at least glancing at [this overview of K8 tunneling options](https://openziti.io/docs/reference/tunnelers/kubernetes/) (Ziti proxy and reverse proxy alternatives for K8s).

I would probably use the node proxy daemonset for both proxy and reverse proxy if all pods in each cluster should have the same level of access. If pods need access to specific, different Ziti services then you must use the pod sidecar proxy as a Ziti client.

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [June 4, 2024, 12:36pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/18 "2024-06-04T12:36:34Z")

</div>

@qrkourier for the same case what would be the bootstrap.env ? if router deployed as a service in ubuntu machine in private subnet and want to be accessible with tunnels on other vpc's

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [June 4, 2024, 12:40pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/19 "2024-06-04T12:40:37Z")

</div>

The main environment variables for generating a Linux router (same for Docker router) configuration are the controller's address and the router's address, and the router's token.

Here's the "bootstrapping" section from [the Linux guide](https://openziti.io/docs/guides/deployments/linux/router/deploy) which mentions the most relevant variables that you should set before starting the Linux service.

At a minimum, you must set these to opt-in to bootstrapping.

1. In **/opt/openziti/etc/router/service.env** (the answer file for the service)
  1. Set `ZITI_BOOTSTRAP=true`

2. In **/opt/openziti/etc/router/bootstrap.env** (the answer file for bootstrapping the router)
  1. Set `ZITI_CTRL_ADVERTISED_ADDRESS` to the FQDN of the controller
  2. Set `ZITI_ENROLL_TOKEN` to the resulting token (JWT)

Additionally, you probably want to set these:

1. In **/opt/openziti/etc/router/bootstrap.env** (the answer file for bootstrapping the router)
  1. Set `ZITI_ROUTER_ADVERTISED_ADDRESS` to the permanent FQDN of the router (default: localhost). This value can not be changed after enrollment.
  2. Set `ZITI_ROUTER_MODE` (default: none) if this router's built-in tunneler will provide a proxy for services. Mode `tproxy` requires additional kernel capabilities in **/etc/systemd/system/ziti-router.service.d/override.conf** and DNS resolver configuration for the host.

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [June 4, 2024, 12:43pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/20 "2024-06-04T12:43:36Z")

</div>

ZITI\_CTRL\_ADVERTISED\_ADDRESS--\> I set to exactly the controller's **ctrl.endpoint="{{ ctrlPlane.advertisedHost }}:6262"** and router's address using internal IP **ZITI\_ROUTER\_ADVERTISED\_ADDRESS=router1.10.0.11.171**

But its not working , the requests from the tunneler are not able to reach the router

**ERROR ziti-sdk:channel.c:903 on\_channel\_connect\_internal() ch[0] failed to connect to ER[router1] [-30**

and then i tried to access from a machine outside of this VPC

---

<div class="post-metadata">

**Author:** ![sadath-12](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/sadath-12/32/2043_2.png) [@sadath-12](https://openziti.discourse.group/u/sadath-12)\
**Post date:** [June 4, 2024, 12:59pm UTC](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616/21 "2024-06-04T12:59:08Z")

</div>

@qrkourier provided this time I have only one private router

[Next page](https://openziti.discourse.group/t/what-would-openziti-arch-for-this-look-like/2616.md?page=2)
