Setup new edge router on a separate host


Following quickstart, I have the controller up and running on a host. I also setup an edge router on a different host following quickstart. What do I need to do in order to have the router to talk to the controller on the first host? I can imagine that I need to somehow “merge” the pki of the two hosts and modify the edge router yaml file. A checklist would be greatly helpful.


Yeah, we need to make that guide. We keep making the doc better but it’s a process that just takes a while. In the meantime, I’ll make that punchlist of things to do and follow up in a bit.

Awesome! I’ll be waiting for it.

Here’s a video outlining the process. We are close to having a helper function that does this, it’s just not released yet. Should be soon. Till then, here you go. Hope that helps

Steps performed:

  1. get the external address of the controller from the config file

  2. make sure the address is addressable from edge router 2 (ER2 from hereon). Make sure curl returns some json.

    curl -sk
  3. determine the control plane port:

    grep ZITI_CTRL_PORT $ZITI_HOME/$(hostname).env
    export ZITI_CTRL_PORT="8440"
  4. make sure ER2 can see the link plane port

    curl -vsk -m 5
    verify you don't see: "* Connection timed out after 5000 milliseconds"
  5. make sure the controller and edge-router-1 are running

    curl -vsk -m 5
    verify you don't see: "* Connection timed out after 5000 milliseconds"
  6. use getZiti to get the binaries (pass ‘yes’ to add ziti to your path)

    source /dev/stdin <<< "$(wget -qO-"; getZiti yes
  7. login to the controller. easier if you do this from ER2, but not necessary (use the value from above)

    ziti edge login
    or `zitiLogin` if you're on the controller
  8. create the ER2 in the controller.

    ziti edge create edge-router -t ER2 -o $ZITI_HOME/ER2.jwt
  9. set some variables, then decide if this is a “private” edge router or public (see below)

    export ROUTER_NAME=ER2
    export ZITI_CTRL_PORT=8440
    # optional - if you want to allow addressing by some specific IP
  10. create a config file. this makes the router ‘private’, meaning it won’t advertise that routers should link to it.

    ziti create config router edge --routerName  $ROUTER_NAME \
                                --output $ZITI_HOME/$ROUTER_NAME.yaml \
                                --tunnelerMode host \
  11. enroll the router (does not matter which config you use, i chose the private one here):

    ziti-router enroll $ZITI_HOME/$ROUTER_NAME.yaml --jwt $ZITI_HOME/ER2.jwt
  12. run the public or private ER2 config:

    ziti-router run $ZITI_HOME/$ROUTER_NAME.yaml
  13. make sure ER2 shows online

    ziti edge list edge-routers
  14. make sure it linked:

    ziti fabric list links
  15. make systemd unit file:

    createRouterSystemdFile $ROUTER_NAME
  16. systemd file stuff (optional)

    sudo cp /home/ubuntu/.ziti/quickstart/$ROUTER_NAME.service /etc/systemd/system/ziti-router.service
    sudo systemctl daemon-reload
    sudo systemctl enable --now ziti-router
1 Like

thanks so much! will give it a try

Almost there. It failed at step 10:

mango@kyoto:~/tools$ ziti create config router edge --routerName er2 --output $ZITI_HOME/er2.yaml --tunnelerMode host
2023/01/17 11:18:34 if ZITI_EDGE_ROUTER_ADVERTISED_HOST[kyoto] is supplied, it *MUST* match the ZITI_EDGE_ROUTER_IP_OVERRIDE[] or resolved hostname[kyoto-edge-router]
panic: if ZITI_EDGE_ROUTER_ADVERTISED_HOST[kyoto] is supplied, it *MUST* match the ZITI_EDGE_ROUTER_IP_OVERRIDE[] or resolved hostname[kyoto-edge-router]

goroutine 1 [running]:
log.Panicf({0x2ff9ab2?, 0x11?}, {0xc00098fcc8?, 0xc000b08600?, 0x0?})
        log/log.go:395 +0x67, {0x7fff8d721af0, 0x3}) +0x408, {0x2f0b59d?, 0x6?, 0x6?}) +0x105*Command).execute(0xc0009b1200, {0xc000af6a20, 0x6, 0x6}) +0x711*Command).ExecuteC(0x480aca0) +0x3bd*Command).Execute(...) +0x38
main.main() +0x17

Man, that’s one of the reasons it took me a while, and why I recorded the video, so that I knew it worked. :slight_smile: Did you perhaps run this on the machine where the quickstart was run, or did you source the .env file from the output of the quickstart? I didn’t do it that way when I tested (I’ll try now), I suspect that might be why you hit this. If not, looks like you might be hitting a bug. I am hitting that same bug with a different discourse post.

I’ll work up alternate instructions by emitting the config and editing it.

oh – that’s my fault! I’ve updated step 10. This can’t be run. You can fix that by running unset ZITI_EDGE_ROUTER_ADVERTISED_HOST

Then the other steps should work

One step closer. Step 12:

mango@kyoto:~/tools$ ziti-router run $ZITI_HOME/er2.yaml
[   0.776]    INFO ziti/ziti/ {routerId=[wCyoZGsOKT] revision=[7de87562b20d] version=[v0.27.2] go-version=[go1.19.4] os=[linux] build-date=[2023-01-04T20:13:07Z] arch=[amd64] configFile=[/home/mango/.ziti/quickstart/kyoto/er2.yaml]} starting ziti-router
[   0.777]    INFO fabric/router/forwarder.(*Faulter).run: started
[   0.777]    INFO fabric/router/forwarder.(*Scanner).run: started
[   0.778]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[] minWorkers=[0] maxWorkers=[32] idleTime=[30s] maxQueueSize=[1000]} starting goroutine pool
[   0.778]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[128] idleTime=[30s] poolType=[pool.route.handler] maxQueueSize=[1000] minWorkers=[0]} starting goroutine pool
[   0.778] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[   0.779]    INFO fabric/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":1000000000,"DelayRxStart":false,"WriteTimeout":0}
[   0.779]    INFO fabric/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"MessageQueueSize":10}
[   0.779]    INFO fabric/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[   0.779]    INFO fabric/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[   0.779]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[10s] maxQueueSize=[1] minWorkers=[1] poolType=[] maxWorkers=[16]} starting goroutine pool
[   0.779]    INFO fabric/router.(*Router).startXlinkListeners: started Xlink listener with binding [transport] advertising [tls:kyoto-edge-router:10080]
[   0.779]    INFO edge/router/xgress_edge.(*listener).Listen: {address=[tls:]} starting channel listener
[   0.780]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] maxWorkers=[16] maxQueueSize=[1] poolType=[pool.listener.xgress_edge] idleTime=[10s]} starting goroutine pool
[   0.780]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:]
[   0.780]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[   0.780]    INFO edge/router/xgress_edge.(*Acceptor).Run: starting
[   0.780]   FATAL ziti/ziti/ {error=[error connecting ctrl (dial tcp connect: connection refused)]} error starting

I guess it’s trying to connect to the controller on the localhost while the actual controller is on a different host.

It is, yes. For this, it will probably be easier to just edit the config file directly. If you open $ZITI_HOME/er2.yaml it should have a section like this:

  endpoint:             tls:ip-172-31-45-134:6262

You want that endpoint to be the host and port of your controller. If you compare the edge router the quickstart created it should be really clear. For example, my “host it anywhere” router config looks like this:


worked! This is on top of an existing quickstart installation. I’ll try a blank host and see if I can repeat the success.

Great to hear! Let me/us know on the next one. Like anything, it gets easier each time you do it.

1 Like