Ziti Edge Router General Questions

Hello. Iโ€™m just expanding my knowledge about Ziti stuff, and I was looking into the Edge Routers, and I have some questions.

From what I can understand, Edge Routers help with, well, routing, making it so better optimized routes can be made between clients. They also help with availability, more public routers, the services will have more availability in case one of the routers fails.

Reading the docs I stumbled upon this page and I was wondering, what is the difference between a private and public edge router?

In the setup Iโ€™m planning, I will have a public controller/router on a remote Oracle Cloud machine. Some of the services I will be hosting will be at my home, and sometimes I will access those services through OpenZiti so I was thinking if it makes sense to setup a edge router inside my network, so this way the packets donโ€™t need to exit my own network and it can be somewhat faster (I assume). Does this make sense? Should I deploy a public or a private edge router?

And as a final question, are there any tutorials on how to deploy these new edge routers with Docker Compose? In the quickstart I see that the edge router and the controller are deployed together and the edge router has access to the same files that the controller has, but in a distributed environment, how would I setup this? What do I need to generate/deliver to the new edge routers?

Thanks in advance for your time :slight_smile:

A recent discourse post is highly relevant for this question. They happened to be using satellite internet and wanted to make sure local traffic was secure without having to rendezvous off the local network (sound familiar?!? :slight_smile: ). See this discourse post for the whole thread.

The most relevant bit to me was the basic design:

So with this setup, they can access all their local network machine via OpenZiti as well and not have to pay the latency price of leaving the home network (which is probably way higher with satellite than either you or I experience). Also, zero trust extends even onto your home network imo! Well, it should anyway, particularly if youโ€™re adding any IoT type devices. So yes, what youโ€™re planning makes perfect sense indeed.

Iโ€™ll give this a run now and make sure I can connect two routers together using nothing but docker/compose! :slight_smile: Iโ€™ll post back in a bit.

1 Like

How lucky! Iโ€™ll make sure to give it a thorough read!

That sounds fantastic, thanks a ton! :slight_smile:

I ran into this awhile ago. There is now a script in the quickstart to address this: run-extrouter.sh. The problem with the quickstart is it requires a configured ziti.env file to be copied from the controller setup to each edge router, which make this problematic for multiple host deployment. So, a script was submitted to remove this requirement and pass everything through the environment variables. The motivation was to create a host with controller/console and another host providing the edge router.

This is a modifed docker-compose file that I use to provide just the edge router component

version: '2.4'
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
      - ZITI_EDGE_ROUTER_RAWNAME=oracle-cloud-router
      - "443:3022"
      - zitiblue
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-extrouter.sh edge"

    driver: bridge

I also have a .env defined as follows:

# OpenZiti Variables

# The duration of the enrollment period (in minutes), default if not set

## Additional variables to override.

Here is the reference to the PR which contains variables etc: Adding ability to run external router by gooseleggs ยท Pull Request #966 ยท openziti/ziti (github.com)

Note that I have set up my environment to only talk on 80/443 so not showing as default values.

@gooseleggs This seems exactly what I was looking for, thank you!

Yep I noticed, nice touch :slight_smile:

A couple questions though

  1. The zitiblue network is not required right? Seems like remnants of the original docker compose quickstart
  2. Does this run-extrouter.sh script also do the configuration required to automatically set it in such a way that this router will be preferred by devices on the same network as this router? I was taking a look at the thread that @TheLumberjack mentioned, and it seemed like at some point you have to enter the console to add certain special policies, or am I mistaken?

Again, thanks for all the help :bowing_man:

Correct about ziti blue. That is me being lazy with config. All the script does is create an id and join it to the fabric ready to do stuff. I havenโ€™t looked at the thread referenced as not needed to do yet but will at some point.

That script is great! I wish I remembered it and looked at it before I went off on the tangent I did todayโ€ฆ :slight_smile: I didnโ€™t see where the ziti_user/pwd were set in your example @gooseleggs?

@jruiz94 thereโ€™s an alternate approach if you want/need itโ€ฆ @gberl002 has some changes still pending that would hopefully make this process easier but truthfully, I didnโ€™t find the process particularly ergonomic yet. I think we have a lot of room to make this better. Hereโ€™s the process I cooked up if youโ€™re interestedโ€ฆ There are new containers that @qrkourier published but they are more โ€œbare bonesโ€, meaning you give it a folder representing an enrolled router and itโ€™ll just let you start/stop it.

I added all the steps directly to a gist. You can grab it from github here, look it over and see what it does and where to modify it. I tried to make it easy, but I really want to work on the ergonomics of this process.

You can read the commands that are in that file if youโ€™re interested. At the end of the day it will:

  • emit a file with some environment variables
  • emit a docker-compose.yaml file to use
  • call the addRouter function to make the new router and enroll it
  • sed the file for use inside docker
  • call docker compose down/up to kick it off and prove it works

Hereโ€™s me demonstrating that it works โ€œfor meโ€ - remember youโ€™d need to modify the env file a bitโ€ฆ it doesnโ€™t โ€œjust workโ€ for you as is :slight_smile: โ€ฆ Also note that I source in my ZITI_USER and ZITI_PWD right at the beginningโ€ฆ The commands I ran, in case itโ€™s too small to see:

  • source ziti-cred to get ZITI_USER and ZITI_PWD into my env vars
  • wget https://gist.githubusercontent.com/dovholuknf/2637742f34ec7b885ed921de9eeaf2ea/raw/d74719a37ed716bc6422ff978b6fbdba54d6c7eb/quick-docker-compose-router.sh
  • bash quick-docker-compose-router.sh newest-er-local 4400 /home/cd/easy-router/newest to run my gist for this environment, making a new router named โ€œnewest-er-localโ€ that listens for edge connections on port 4400


@TheLumberjack - I probably passed the ZITI_USER and ZITI_PWD in through the environment variables, but once created, deleted them from the docker file. I need to stand up another instance of it in OCI, so I could make a how-to if that any use.

I am not sure where the future of the bare bones containers will lie. Would be good to move towards removing the quickstart containers to be a docker-compose file using the separate components (and the appropriate scripts included). That said, those separate images can get quite large compared to the one quickstart image.

@jruiz94 - while I think about it, if you use the docker-compose script - start it up in normal mode docker-compose up first and it will tell you if you are missing anything and see it step through the actions. When happy, then you can remove most of the variables and run it is daemon mode.

Thanks for sharing the gist! Although Iโ€™m going to give @gooseleggs 's solution a shot first since it seems pretty straightforward. However Iโ€™ve read through the gist and Itโ€™s really useful in order to see the full sequence of steps to have it present in my head. So kudos for that! :smiley:

Sure Iโ€™ll double check everything, thanks for the heads up :slight_smile:

One question I have for both though, when I finally set this up, is there a way that I can verify that the routing is being performed correctly? AKA the request is going from my computer, to the edge router at home (instead of the cloud edge router), to its destination at home.

Only sure-fire way is to look at your traffic using something like tccpdump. Per that other discourse post, youโ€™ll see toward the end we needed to figure out why the traffic was going out to the public router only to come back in again. The issue was the config file didnโ€™t have the proper advertised url in itโ€ฆ So local clients couldnโ€™t connect.

But, you can verify it with ziti by making a connection to something, ssh for example, then find the circuit youโ€™re connecting with. Using commands like fabric list circuts, edge list sessions, edge list api-sessions:

root@ip-172-31-11-231 ~
$ ziti fabric list circuits
โ”‚ ID        โ”‚ CLIENT                    โ”‚ SERVICE      โ”‚ TERMINATOR             โ”‚ PATH                           โ”‚
โ”‚ LyXEFOIuE โ”‚ clh0nr8mx04wqabqfa1qxnj6f โ”‚ g34688rhljci โ”‚ 1egmu3usNESJ3yVRRfv4Bg โ”‚ r/ip-172-31-11-231-edge-router โ”‚
results: 1-1 of 1

root@ip-172-31-11-231 ~
$ ziti edge list sessions 'id = "clh0nr8mx04wqabqfa1qxnj6f"'
โ”‚ ID                        โ”‚ API SESSION ID            โ”‚ SERVICE NAME โ”‚ TYPE โ”‚
โ”‚ clh0nr8mx04wqabqfa1qxnj6f โ”‚ clh0nq37v04uvabqfz776s8gu โ”‚ g34688rhljci โ”‚ Dial โ”‚
results: 1-1 of 1

root@ip-172-31-11-231 ~
$ ziti edge list api-sessions 'id = "clh0nq37v04uvabqfz776s8gu"'
โ”‚ ID                        โ”‚ TOKEN                                โ”‚ IDENTITY NAME โ”‚
โ”‚ clh0nq37v04uvabqfz776s8gu โ”‚ c17e6b77-8eff-42d4-82be-492a5e52c157 โ”‚ frontend      โ”‚
results: 1-1 of 1

Here you can see my client was โ€˜frontendโ€™ and it used the router r/ip-172-31-11-231-edge-router. If it went through more than one router you would see more than one router in the circuit and you would see what routers were traversed.

1 Like

Thanks! Iโ€™ll try to verify when I have everything set up :slight_smile:

Another question, in order to host my private Ziti Edge Router, do I need to open and forward the 3022 port (or whatever port iโ€™d like to use) in my router and assign a static ip to the machine running this docker compose? Reading the examples you gave me, it seems so, but I just wanted to be sure.

Yes. The edge router will need whatever port is advertised to be open, allowing clients to establish mTLS connections to the router. A static IP is only needed if you donโ€™t have a DNS entry, but that is vital too. Once enrolled, the IP and DNS shouldnโ€™t change or the pki will most likely become invalid.

Hmmm I see. For my private LAN IP thatโ€™s no problem, I can set up a static IP for the device.

For the public IPโ€ฆ where I live, the ISP only guarantees your IP as long as your router doesnโ€™t go down. If the router turns off then on, the IP might change.

Some years ago when I set up wireguard for my homelab I solved this by using duckdns and a cron script to automatically update each 5 min the DNS entry pointing to whatever my current IP was at the moment.

I wonโ€™t be using duckdns this time but I bet thereโ€™s some way to implement a similar thing where Iโ€™m constantly updating the dns entry with the current IP whatever it might be.

If Iโ€™m doing this to keep the DNS entry updated, would that work for the OpenZiti Edge Router? Or is it forced to have always the same public IP?

I recommend using something like Oracleโ€™s free tier for public access. Itโ€™ll be a static IP forever, I think. Then your private router links into that. I donโ€™t recommend relying on your isp ip in that way. Plus you wonโ€™t have to open holes in your firewall this way.

Yes, I would expect that would work out just fine if you use DNS instead of ip. Itโ€™s why we tend to try to favor DNS instead of ip.

Wait I might have misunderstood then, so for the private edge router thereโ€™s no need to open ports in my routerโ€™s firewall? Will it โ€œlinkโ€ to the public one?

Yes. Exactly. The private router will be configured with a link โ€œdialerโ€. The public router is configured with a link dialer AND a link โ€œlistenerโ€. The listener is the only port that needs to be open. Itโ€™s a different port too than the one you probably opened already in the public router so double check the link.listener of the public router.

Keep your home firewall 100% closed! :sunglasses:

1 Like

@gooseleggs I tried to follow your script. I have a couple questions/notes for future reference

The script is now called run-router-external.sh apparently

As you predicted, I got a couple errors about variables undefined which I tried to fix by filling them out, but then I ended up with a weird error:
Unable to connect to the server: dial tcp i/o timeout
The is a fake ip im making up to refer to the IP where I have the oracle cloud instance with the public controller/router., but I donโ€™t understand why it does try to connect using port 443, I thought I was specifying the correct port (I made sure to remove all references to port 443 from my docker compose)

But overall, I find it a little difficult to debug and understand what are the expected values for the environment variables, because either Iโ€™m not understanding the correct naming for each Ziti element or I find the naming a little bit inconsistent.


  1. ZITI_CONTROLLER_HOSTNAME and ZITI_EDGE_CONTROLLER_HOSTNAME. Is an โ€˜Edge Controllerโ€™ different from a plain โ€˜Controllerโ€™? Are they different elements? In my setup I have only one Controller, the one deployed in the OCL instance with the Edge Router, is that an โ€œedgeโ€ or a โ€œregularโ€ controller?

  2. What is the difference between a RAWNAME and a HOSTNAME? I get the hostname is the url where to find the controller/router (in my case mymachine.mydomain.com) but what is this rawname? Is it the same? In which scenarios it isnโ€™t?

  3. Then there are some variables that abbreviate controller for ctrl but some seem duplicates? For example isnโ€™t ZITI_CTRL_ADVERTISED_ADDRESS the same as ZITI_EDGE_CONTROLLER_HOSTNAME or ZITI_CONTROLLER_HOSTNAME?

  4. Thereโ€™s also ZITI_CTRL_PORT ZITI_CTRL_EDGE_ADVERTISED_PORT and ZITI_CTRL_LISTENER_PORT. Iโ€™m using all default so I assume the 2 last ones are 8441 and 8440 respectively, but the first got me confused

  5. Also @gooseleggs , in your .env file you have duplicated ZITI_EDGE_CONTROLLER_RAWNAME i assume the good one is the last one xD

Iโ€™m going to try @TheLumberjack 's approach, itโ€™s a little bit less ergonomic as he said xD But the step-by-step nature of it might be of help in understanding everything better. Although Iโ€™m still a little bit confused with the environment variables that I mentioned beforeโ€ฆ (For example, what is the expected value for these variables ZITI_EDGE_ROUTER_RAWNAME and ZITI_EDGE_ROUTER_PORT when this router is going to be private? I assume port = 3022 and rawname = some name i invent for the router?)

Yeahโ€ฆ The variable names suffer from years of just patching it together to make it workโ€ฆ Geoff has a whole branch where hopefully theyโ€™re cleaned up and much more obvious to people.


The first one is what you want the router to be known by in the controller. So โ€œprivate-routerโ€, or whatever. The second is the port that local clients will connect to the edge router at. So that port needs to be open to private network clients (not externally), but like you said that should not be a problem.

Make sure you read and adjust the gist and then you should be able to just execute itโ€ฆ Probably. This is clearly an area we have for improvement. :slight_smile:

Hopefully itโ€™s enough to get your private router running

1 Like

@TheLumberjack Quick question, I just realized that, although the Oracle Cloud public instance has the following ports open (from the Minecraft tutorial)
since Iโ€™ve been using the docker compose quickstart, maybe Iโ€™m not exposing all the ports that are needed to do this? These are the only ports exposed in the controller and the public router:

    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
      - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
    . . .
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
   . . .

Iโ€™ve been trying both your solution and @gooseleggs but I havenโ€™t had success so far, could it be that Iโ€™m making a huge dumb mistake by not mapping the ports 8441 and 8440 in my docker compose? :man_facepalming: Also maybe I need to map the port 10080 too?

Your public router would need to set and expose one additional port, 10080. Thatโ€™s the one that routers will form links over.

There are generally up to five ports that need to be exposed. The ports shown below are โ€œdefaultโ€/โ€œquickstartโ€:

Ziti Controller:

  • API port (the port that remote clients use to talk to the controller) - often 1280/8441
  • โ€œControl planeโ€ port (the port routers use to talk to the controller) - often 6262/8440

Ziti Routers:

  • Edge port (the port that edge clients use to establish the data plane) - often 3022/8442
  • Fabric port (the port edge routers will connect to other edge routers over) - often 10080/NA


  • non TLS (http) 1408/1408
  • TLS 8443/8443

So, if you want to have your private router link to your public edge router, whatever value is in the public edge routerโ€™s โ€œlink.listeners.bindโ€ is what you need to have open in order to allow the private edge router to dial out and make that link.

Letโ€™s take my example. I have deployed out to AWS and used dockerโ€ฆ I execโ€™ed into the container and looked at my router config. It shows me this:

    - binding: transport
    - binding:          transport
      bind:             tls:
      advertise:        tls:ec2-3-134-108-218.us-east-2.compute.amazonaws.com:8444
        outQueueSize:   4

This is what is relayed to routers when they come online.Since I have a โ€œlink.listenerโ€ defined here, other routers will try to form a TLS link to ec2-3-134-108-218.us-east-2.compute.amazonaws.com:8444 (you can see I changed the port and used a NEW one 8444, not 10080) :slight_smile:

Make sense?