Hello. I’m just expanding my knowledge about Ziti stuff, and I was looking into the Edge Routers, and I have some questions.
From what I can understand, Edge Routers help with, well, routing, making it so better optimized routes can be made between clients. They also help with availability, more public routers, the services will have more availability in case one of the routers fails.
Reading the docs I stumbled upon this page and I was wondering, what is the difference between a private and public edge router?
In the setup I’m planning, I will have a public controller/router on a remote Oracle Cloud machine. Some of the services I will be hosting will be at my home, and sometimes I will access those services through OpenZiti so I was thinking if it makes sense to setup a edge router inside my network, so this way the packets don’t need to exit my own network and it can be somewhat faster (I assume). Does this make sense? Should I deploy a public or a private edge router?
And as a final question, are there any tutorials on how to deploy these new edge routers with Docker Compose? In the quickstart I see that the edge router and the controller are deployed together and the edge router has access to the same files that the controller has, but in a distributed environment, how would I setup this? What do I need to generate/deliver to the new edge routers?
A recent discourse post is highly relevant for this question. They happened to be using satellite internet and wanted to make sure local traffic was secure without having to rendezvous off the local network (sound familiar?!? ). See this discourse post for the whole thread.
The most relevant bit to me was the basic design:
So with this setup, they can access all their local network machine via OpenZiti as well and not have to pay the latency price of leaving the home network (which is probably way higher with satellite than either you or I experience). Also, zero trust extends even onto your home network imo! Well, it should anyway, particularly if you're adding any IoT type devices. So yes, what you're planning makes perfect sense indeed.
I'll give this a run now and make sure I can connect two routers together using nothing but docker/compose! I'll post back in a bit.
I ran into this awhile ago. There is now a script in the quickstart to address this: run-extrouter.sh. The problem with the quickstart is it requires a configured ziti.env file to be copied from the controller setup to each edge router, which make this problematic for multiple host deployment. So, a script was submitted to remove this requirement and pass everything through the environment variables. The motivation was to create a host with controller/console and another host providing the edge router.
This is a modifed docker-compose file that I use to provide just the edge router component
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest
ZITI_CONTROLLER_RAWNAME=ziticontroller
ZITI_EDGE_CONTROLLER_RAWNAME=ziti-edge-controller
# The duration of the enrollment period (in minutes), default if not set
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=
ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=
## Additional variables to override.
ZITI_EDGE_CONTROLLER_RAWNAME=ziticontroller.example.com
ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT=ziticontroller.example.com:443
ZITI_EDGE_CONTROLLER_HOSTNAME=ziticontroller.example.com
ZITI_CTRL_ADVERTISED_ADDRESS=ziticontroller.example.com
ZITI_CONTROLLER_HOSTNAME=ziticontroller.example.com
ZITI_CTRL_PORT=80
@gooseleggs This seems exactly what I was looking for, thank you!
Yep I noticed, nice touch
A couple questions though
The zitiblue network is not required right? Seems like remnants of the original docker compose quickstart
Does this run-extrouter.sh script also do the configuration required to automatically set it in such a way that this router will be preferred by devices on the same network as this router? I was taking a look at the thread that @TheLumberjack mentioned, and it seemed like at some point you have to enter the console to add certain special policies, or am I mistaken?
Correct about ziti blue. That is me being lazy with config. All the script does is create an id and join it to the fabric ready to do stuff. I haven’t looked at the thread referenced as not needed to do yet but will at some point.
That script is great! I wish I remembered it and looked at it before I went off on the tangent I did today… I didn’t see where the ziti_user/pwd were set in your example @gooseleggs?
@jruiz94 there’s an alternate approach if you want/need it… @gberl002 has some changes still pending that would hopefully make this process easier but truthfully, I didn’t find the process particularly ergonomic yet. I think we have a lot of room to make this better. Here’s the process I cooked up if you’re interested… There are new containers that @qrkourier published but they are more “bare bones”, meaning you give it a folder representing an enrolled router and it’ll just let you start/stop it.
I added all the steps directly to a gist. You can grab it from github here, look it over and see what it does and where to modify it. I tried to make it easy, but I really want to work on the ergonomics of this process.
You can read the commands that are in that file if you’re interested. At the end of the day it will:
emit a file with some environment variables
emit a docker-compose.yaml file to use
call the addRouter function to make the new router and enroll it
sed the file for use inside docker
call docker compose down/up to kick it off and prove it works
Here’s me demonstrating that it works “for me” - remember you’d need to modify the env file a bit… it doesn’t “just work” for you as is … Also note that I source in my ZITI_USER and ZITI_PWD right at the beginning… The commands I ran, in case it’s too small to see:
source ziti-cred to get ZITI_USER and ZITI_PWD into my env vars
bash quick-docker-compose-router.sh newest-er-local 4400 /home/cd/easy-router/newest to run my gist for this environment, making a new router named “newest-er-local” that listens for edge connections on port 4400
@TheLumberjack - I probably passed the ZITI_USER and ZITI_PWD in through the environment variables, but once created, deleted them from the docker file. I need to stand up another instance of it in OCI, so I could make a how-to if that any use.
I am not sure where the future of the bare bones containers will lie. Would be good to move towards removing the quickstart containers to be a docker-compose file using the separate components (and the appropriate scripts included). That said, those separate images can get quite large compared to the one quickstart image.
@jruiz94 - while I think about it, if you use the docker-compose script - start it up in normal mode docker-compose up first and it will tell you if you are missing anything and see it step through the actions. When happy, then you can remove most of the variables and run it is daemon mode.
Thanks for sharing the gist! Although I'm going to give @gooseleggs 's solution a shot first since it seems pretty straightforward. However I've read through the gist and It's really useful in order to see the full sequence of steps to have it present in my head. So kudos for that!
Sure I'll double check everything, thanks for the heads up
One question I have for both though, when I finally set this up, is there a way that I can verify that the routing is being performed correctly? AKA the request is going from my computer, to the edge router at home (instead of the cloud edge router), to its destination at home.
Only sure-fire way is to look at your traffic using something like tccpdump. Per that other discourse post, you'll see toward the end we needed to figure out why the traffic was going out to the public router only to come back in again. The issue was the config file didn't have the proper advertised url in it... So local clients couldn't connect.
But, you can verify it with ziti by making a connection to something, ssh for example, then find the circuit you're connecting with. Using commands like fabric list circuts, edge list sessions, edge list api-sessions:
root@ip-172-31-11-231 ~
$ ziti fabric list circuits
╭───────────┬───────────────────────────┬──────────────┬────────────────────────┬────────────────────────────────╮
│ ID │ CLIENT │ SERVICE │ TERMINATOR │ PATH │
├───────────┼───────────────────────────┼──────────────┼────────────────────────┼────────────────────────────────┤
│ LyXEFOIuE │ clh0nr8mx04wqabqfa1qxnj6f │ g34688rhljci │ 1egmu3usNESJ3yVRRfv4Bg │ r/ip-172-31-11-231-edge-router │
╰───────────┴───────────────────────────┴──────────────┴────────────────────────┴────────────────────────────────╯
results: 1-1 of 1
root@ip-172-31-11-231 ~
$ ziti edge list sessions 'id = "clh0nr8mx04wqabqfa1qxnj6f"'
╭───────────────────────────┬───────────────────────────┬──────────────┬──────╮
│ ID │ API SESSION ID │ SERVICE NAME │ TYPE │
├───────────────────────────┼───────────────────────────┼──────────────┼──────┤
│ clh0nr8mx04wqabqfa1qxnj6f │ clh0nq37v04uvabqfz776s8gu │ g34688rhljci │ Dial │
╰───────────────────────────┴───────────────────────────┴──────────────┴──────╯
results: 1-1 of 1
root@ip-172-31-11-231 ~
$ ziti edge list api-sessions 'id = "clh0nq37v04uvabqfz776s8gu"'
╭───────────────────────────┬──────────────────────────────────────┬───────────────╮
│ ID │ TOKEN │ IDENTITY NAME │
├───────────────────────────┼──────────────────────────────────────┼───────────────┤
│ clh0nq37v04uvabqfz776s8gu │ c17e6b77-8eff-42d4-82be-492a5e52c157 │ frontend │
╰───────────────────────────┴──────────────────────────────────────┴───────────────╯
results: 1-1 of 1
Here you can see my client was 'frontend' and it used the router r/ip-172-31-11-231-edge-router. If it went through more than one router you would see more than one router in the circuit and you would see what routers were traversed.
Thanks! I'll try to verify when I have everything set up
Another question, in order to host my private Ziti Edge Router, do I need to open and forward the 3022 port (or whatever port i'd like to use) in my router and assign a static ip to the machine running this docker compose? Reading the examples you gave me, it seems so, but I just wanted to be sure.
Yes. The edge router will need whatever port is advertised to be open, allowing clients to establish mTLS connections to the router. A static IP is only needed if you don’t have a DNS entry, but that is vital too. Once enrolled, the IP and DNS shouldn’t change or the pki will most likely become invalid.
Hmmm I see. For my private LAN IP that’s no problem, I can set up a static IP for the device.
For the public IP… where I live, the ISP only guarantees your IP as long as your router doesn’t go down. If the router turns off then on, the IP might change.
Some years ago when I set up wireguard for my homelab I solved this by using duckdns and a cron script to automatically update each 5 min the DNS entry pointing to whatever my current IP was at the moment.
I won’t be using duckdns this time but I bet there’s some way to implement a similar thing where I’m constantly updating the dns entry with the current IP whatever it might be.
If I’m doing this to keep the DNS entry updated, would that work for the OpenZiti Edge Router? Or is it forced to have always the same public IP?
I recommend using something like Oracle's free tier for public access. It'll be a static IP forever, I think. Then your private router links into that. I don't recommend relying on your isp ip in that way. Plus you won't have to open holes in your firewall this way.
Yes, I would expect that would work out just fine if you use DNS instead of ip. It's why we tend to try to favor DNS instead of ip.
Wait I might have misunderstood then, so for the private edge router there's no need to open ports in my router's firewall? Will it "link" to the public one?
Yes. Exactly. The private router will be configured with a link “dialer”. The public router is configured with a link dialer AND a link “listener”. The listener is the only port that needs to be open. It’s a different port too than the one you probably opened already in the public router so double check the link.listener of the public router.
@gooseleggs I tried to follow your script. I have a couple questions/notes for future reference
The script is now called run-router-external.sh apparently
As you predicted, I got a couple errors about variables undefined which I tried to fix by filling them out, but then I ended up with a weird error: Unable to connect to the server: dial tcp 11.22.33.44:443: i/o timeout
The 11.22.33.44 is a fake ip im making up to refer to the IP where I have the oracle cloud instance with the public controller/router., but I don't understand why it does try to connect using port 443, I thought I was specifying the correct port (I made sure to remove all references to port 443 from my docker compose)
But overall, I find it a little difficult to debug and understand what are the expected values for the environment variables, because either I'm not understanding the correct naming for each Ziti element or I find the naming a little bit inconsistent.
Examples:
ZITI_CONTROLLER_HOSTNAME and ZITI_EDGE_CONTROLLER_HOSTNAME. Is an 'Edge Controller' different from a plain 'Controller'? Are they different elements? In my setup I have only one Controller, the one deployed in the OCL instance with the Edge Router, is that an "edge" or a "regular" controller?
What is the difference between a RAWNAME and a HOSTNAME? I get the hostname is the url where to find the controller/router (in my case mymachine.mydomain.com) but what is this rawname? Is it the same? In which scenarios it isn't?
Then there are some variables that abbreviate controller for ctrl but some seem duplicates? For example isn't ZITI_CTRL_ADVERTISED_ADDRESS the same as ZITI_EDGE_CONTROLLER_HOSTNAME or ZITI_CONTROLLER_HOSTNAME?
There's also ZITI_CTRL_PORTZITI_CTRL_EDGE_ADVERTISED_PORT and ZITI_CTRL_LISTENER_PORT. I'm using all default so I assume the 2 last ones are 8441 and 8440 respectively, but the first got me confused
Also @gooseleggs , in your .env file you have duplicated ZITI_EDGE_CONTROLLER_RAWNAME i assume the good one is the last one xD
I'm going to try @TheLumberjack 's approach, it's a little bit less ergonomic as he said xD But the step-by-step nature of it might be of help in understanding everything better. Although I'm still a little bit confused with the environment variables that I mentioned before... (For example, what is the expected value for these variables ZITI_EDGE_ROUTER_RAWNAME and ZITI_EDGE_ROUTER_PORT when this router is going to be private? I assume port = 3022 and rawname = some name i invent for the router?)
Yeah… The variable names suffer from years of just patching it together to make it work… Geoff has a whole branch where hopefully they’re cleaned up and much more obvious to people.
ZITI_EDGE_ROUTER_RAWNAME and ZITI_EDGE_ROUTER_PORT…
The first one is what you want the router to be known by in the controller. So “private-router”, or whatever. The second is the port that local clients will connect to the edge router at. So that port needs to be open to private network clients (not externally), but like you said that should not be a problem.
Make sure you read and adjust the gist and then you should be able to just execute it… Probably. This is clearly an area we have for improvement.
Hopefully it’s enough to get your private router running
@TheLumberjack Quick question, I just realized that, although the Oracle Cloud public instance has the following ports open (from the Minecraft tutorial)
since I’ve been using the docker compose quickstart, maybe I’m not exposing all the ports that are needed to do this? These are the only ports exposed in the controller and the public router:
I’ve been trying both your solution and @gooseleggs but I haven’t had success so far, could it be that I’m making a huge dumb mistake by not mapping the ports 8441 and 8440 in my docker compose? Also maybe I need to map the port 10080 too?
Your public router would need to set and expose one additional port, 10080. That’s the one that routers will form links over.
There are generally up to five ports that need to be exposed. The ports shown below are “default”/“quickstart”:
Ziti Controller:
API port (the port that remote clients use to talk to the controller) - often 1280/8441
“Control plane” port (the port routers use to talk to the controller) - often 6262/8440
Ziti Routers:
Edge port (the port that edge clients use to establish the data plane) - often 3022/8442
Fabric port (the port edge routers will connect to other edge routers over) - often 10080/NA
ZAC:
non TLS (http) 1408/1408
TLS 8443/8443
So, if you want to have your private router link to your public edge router, whatever value is in the public edge router’s “link.listeners.bind” is what you need to have open in order to allow the private edge router to dial out and make that link.
Let’s take my example. I have deployed out to AWS and used docker… I exec’ed into the container and looked at my router config. It shows me this:
link:
dialers:
- binding: transport
listeners:
- binding: transport
bind: tls:0.0.0.0:8444
advertise: tls:ec2-3-134-108-218.us-east-2.compute.amazonaws.com:8444
options:
outQueueSize: 4
This is what is relayed to routers when they come online.Since I have a “link.listener” defined here, other routers will try to form a TLS link to ec2-3-134-108-218.us-east-2.compute.amazonaws.com:8444 (you can see I changed the port and used a NEW one 8444, not 10080)
Also maybe I need to map the port 10080 too?