Edge Router different host via docker

Hey

So, I see in another thread around setting up another edge router on another host separate from the controller. I am doing this with docker. I had a bit of a go yesterday and didn’t get really far. Basically, on one host (Oracle cloud) I have docker along with Controller and Console. On another Oracle host I want to install an edge router (again on docker). I thought it might have been quick and relatively simple, but appears to not be.

I will let you know the issues that I have as going through it again. However, it appears that not all the environment variables are coming into the docker environment.

Anyway - this is a Has anyone done this, and got a recipe for it question. Otherwise, I am working through it and will no doubt add to this topic.

Can you report what the problem you saw was? It "should" definitely be easy and relatively simple, yes for sure. It's almost certainly a case of "nobody has tried to do it before".

At some point, I think it does become easier to just learn how to edit the config file and accomplish what you want to do. The quickstarts have always been intended to serve as learning vehicles primarily, but if we can extend them into practical administration that feels like a win-win situation.

I've not seen anyone use docker specifically to add a second router. The quickstart does show that it "should" work (it specifically adds a second router) so maybe this is a problem with oracle and "ip only" solutions?

Is it clear what went wrong and how?

There are a number of items. I need to document. I went through the steps for “normal” that you posted in a discourse yesterday and didn’t win. There were a couple of gotchas. When I create a list of issues, I will come back here as well as create githubs.

One of the issues I did have is for some reason it is waiting to connect to “:1280” instead of the router name . I have to remember to blow away the volume as it seems once ziti.env is written it is not overwritten with any new variables on container startup. I am sure it is PEBKEK.

One item I did notice, but haven’t got into it yet is that it appears that in the scripts it does the whole createPki script for running an edge router, but the steps given yesterday is just grab a YML, create a JWT enroll it and go.

Will post more tonight.

That sounds like an environment variable that's not setup properly (or a bug). I'm going to try this out if i can in the next hour-ish.

Just so I can replicate - you're running via docker-compose for controller and initial edge router? Not via the "non-docker" quickstart, correct?

And you're adding another public router on another node. Right? I want to make sure I have the scenario correct.

If I don't post back, I'll keep at it on my am time

I’ve gone through it enough to know that it’s not going to work and as it is, the docker container expects to have access to the exact same volume mount that the controller was mapped to, and for no “good reason”. This is another growing pain that’s held over from like, 2-3 years ago.

I’ll try to rework this in the morning so that it’ll work. It’s going to require a new container to be used the way you want to use it.

For both of our sake, here is how I started the compose file in OCI. I’m not 100% it works, but I figured it might be interesting to see what I did/how.

# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest

# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest

# The duration of the enrollment period (in minutes), default if not set
# shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=10080

# controller address/port information
#ZITI_CONTROLLER_RAWNAME=ziti-controller
#ZITI_CONTROLLER_HOSTNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
#ZITI_CONTROLLER_HOSTNAME=150.136.151.243
ZITI_CTRL_PORT=8440

#ZITI_EDGE_CONTROLLER_RAWNAME=ziti-edge-controller
#ZITI_EDGE_CONTROLLER_HOSTNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
ZITI_EDGE_CONTROLLER_PORT=8441
ZITI_EDGE_CONTROLLER_IP_OVERRIDE=150.136.151.243

# router address/port information
#ZITI_EDGE_ROUTER_RAWNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
ZITI_EDGE_ROUTER_PORT=8442
ZITI_EDGE_ROUTER_IP_OVERRIDE=150.136.151.243

I also get these sorts of errors which shouldn’t be there now:


Creating server cert from ca: ziti-controller-intermediate for ziti-edge-router-1,localhost,127.0.0.1,ziti / 150.136.151.243
Using CA name:  ziti-controller-intermediate
error: cannot locate signer: failed fetching bundle ziti-controller-intermediate within CA ziti-controller-intermediate: failed reading CA private key from file /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: failed reading /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: open /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: no such file or directory
Creating client cert from ca: ziti-controller-intermediate for ziti-edge-router-1,localhost,127.0.0.1,ziti
Using CA name:  ziti-controller-intermediate
error: cannot locate signer: failed fetching bundle ziti-controller-intermediate within CA ziti-controller-intermediate: failed reading CA private key from file /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: failed reading /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: open /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-intermediate.key: no such file or directory

I had to copy the ziti.env file from the original volume mount, scp it from the controller to the router, then modify it:

#export ZITI_EDGE_CTRL_ADVERTISED="ziti-edge-controller:8441"
export ZITI_EDGE_CTRL_ADVERTISED="150.136.151.243:8441"
#export ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT="ziti-edge-controller:8441"
export ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT="150.136.151.243:8441"

Once modified I ran this slightly different (from the quickstart) docker command which still did NOT succeed:

docker run \
  -e ZITI_EDGE_ROUTER_RAWNAME=ziti-edge-router-1 \
  -e ZITI_EDGE_ROUTER_ROLES=public \
  -p 8442:8442 \
  -it \
  --rm \
  -v $HOME/persistent/ziti.env:/persistent/ziti.env \
  openziti/quickstart \
  /var/openziti/scripts/run-router.sh edge

I will follow up in the am and see if I can get a router container to run in a standalone way, like you’re looking for.

I also need to patch the docker run script too. Everytime the router restarts, it would delete/recreate the edge router. That’s “fine” for one-time, quick deployment/learning stuff, but clearly won’t work for you…

see:

Oh wait, my local copy was just out of date or I read it wrong. It looks like we patched that problem a while back… I’ll go to bed now as apparently I’m seeing things :slight_smile:

Correct. I much prefer the docker way for many reasons. So, host 1 has controller and console roles (basically take the docker-compose and strip all but the controller, init container and console config. Host 2 has the edge router role. I have done this for a couple of reasons

  1. Give it a go
  2. The OCI are small machines (free) so not wanting to load one host up

The plan is that this will eventually become only an OCI router for access to SSH and ZAC (may move it back onto host 1). However for the short term will experiment with it as an edge router for on-premise and see how it performs.

OK - I have a worked through and got a working container. It is not pretty, but it is enrolling and looking good (haven’t put anything though it yet but getting two green dots. Here is what I did. Note that this is cut down, and I think that you could keep the container as is it but add in another script to it.

As an aside comment, I think it could be good to work towards only requiring the environment variables required for this but having the ziti.env environment clobber any passed in variables is annoying. For instance, I see that the ZITI_USER and ZITI_PWD is in the ziti.env. For an external router, these variables should only need to be passed through to initiate the config, then should be able to be deleted from the docker-compose file as no longer needed. This make sense? Maybe I could work towards a production container vs a quickstart container?

Anyhow, what follows is what I have done. Will leave it to you to decide how you want to import it or whatnot.

I created a new openziti/quickstart container (named test:latest here). I modified the image/Dockerfile from GIT and added this line:

COPY --chown=ziti run-extrouter.sh "${ZITI_SCRIPTS}/"

at the end of the other copy lines.

The run-extrouter.sh file is basically a modified run-router.sh script and lives in the docker/image folder (dont forget to chmod +x it!)

#!/bin/bash

# give the controller time to ramp up before running if running in docker-compose
sleep 5

. "${ZITI_SCRIPTS}/ziti-cli-functions.sh"

if [[ "${ZITI_CONTROLLER_RAWNAME-}" == "" ]]; then export ZITI_CONTROLLER_RAWNAME="ziti-controller"; fi
if [[ "${ZITI_EDGE_CONTROLLER_RAWNAME-}" == "" ]]; then export ZITI_EDGE_CONTROLLER_RAWNAME="ziti-edge-controller"; fi
if [[ "${ZITI_EDGE_ROUTER_RAWNAME-}" == "" ]]; then
  export ZITI_EDGE_ROUTER_DESIRED_RAWNAME="${ZITI_NETWORK-}-edge-router"
else
  ZITI_EDGE_ROUTER_DESIRED_RAWNAME="${ZITI_EDGE_ROUTER_RAWNAME}"
fi
if [[ "${ZITI_EDGE_ROUTER_PORT-}" == "" ]]; then export ZITI_EDGE_ROUTER_PORT="3022"; fi
if [[ "${ZITI_EDGE_ROUTER_HOSTNAME}" == "" ]]; then export ZITI_EDGE_ROUTER_HOSTNAME="${ZITI_EDGE_ROUTER_RAWNAME}${ZITI_DOMAIN_SUFFIX}"; fi
if [[ "${ZITI_EDGE_ROUTER_ROLES}" == "" ]]; then export ZITI_EDGE_ROUTER_ROLES="${ZITI_EDGE_ROUTER_RAWNAME}"; fi

. ${ZITI_HOME}/ziti.env

echo "ZITI_EDGE_ROUTER_RAWNAME = ${ZITI_EDGE_ROUTER_RAWNAME}"
echo "ZITI_EDGE_ROUTER_HOSTNAME = ${ZITI_EDGE_ROUTER_HOSTNAME}"
echo "ZITI_EDGE_ROUTER_ROLES = ${ZITI_EDGE_ROUTER_ROLES}"
echo "ZITI_EDGE_ROUTER_PORT = ${ZITI_EDGE_ROUTER_PORT}"

# If we dont have a router yaml, then we assume we haven't enrolled, so lets do that
if [ ! -f ${ZITI_EDGE_ROUTER_RAWNAME}.yaml ]; then

  # Login to the cloud controller
  ziti edge login -y ${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT} -u ${ZITI_USER} -p ${ZITI_PWD}

  if [[ "$1" == "edge" ]]; then
    echo "CREATING EDGE ROUTER CONFIG"
    createEdgeRouterConfig "${ZITI_EDGE_ROUTER_RAWNAME}"
  fi
  if [[ "$1" == "wss" ]]; then
    echo "CREATING EDGE ROUTER WSS CONFIG"
    createEdgeRouterWssConfig "${ZITI_EDGE_ROUTER_RAWNAME}"
  fi
  if [[ "$1" == "fabric" ]]; then
    echo "CREATING FABRIC ROUTER CONFIG"
    createFabricRouterConfig "${ZITI_EDGE_ROUTER_RAWNAME}"
  fi
  if [[ "$1" == "private" ]]; then
    echo "CREATING PRIVATE ROUTER CONFIG"
    createPrivateRouterConfig "${ZITI_EDGE_ROUTER_RAWNAME}"
  fi

  echo "----------  Creating edge-router ${ZITI_EDGE_ROUTER_HOSTNAME}...."
  found=$(ziti edge list edge-routers 'name = "'"${ZITI_EDGE_ROUTER_HOSTNAME}"'"' | grep -c "${ZITI_EDGE_ROUTER_HOSTNAME}")
  if [[ found -gt 0 ]]; then
    echo "----------  Found existing edge-router ${ZITI_EDGE_ROUTER_HOSTNAME}...."
  else
    "${ZITI_BIN_DIR}/ziti" edge create edge-router "${ZITI_EDGE_ROUTER_HOSTNAME}" -o "${ZITI_HOME}/${ZITI_EDGE_ROUTER_HOSTNAME}.jwt" -t -a "${ZITI_EDGE_ROUTER_ROLES}"
    sleep 1
    echo "---------- Enrolling edge-router ${ZITI_EDGE_ROUTER_HOSTNAME}...."
  "${ZITI_BIN_DIR}/ziti-router" enroll "${ZITI_HOME}/${ZITI_EDGE_ROUTER_HOSTNAME}.yaml" --jwt "${ZITI_HOME}/${ZITI_EDGE_ROUTER_HOSTNAME}.jwt"
    echo ""
  fi
fi

# Run the router
"${ZITI_BIN_DIR}/ziti-router" run "${ZITI_HOME}/${ZITI_EDGE_ROUTER_RAWNAME}.yaml" > "${ZITI_HOME}/ziti-${ZITI_EDGE_ROUTER_HOSTNAME}.log"

Build the container like normal. Note that the helper functions create pki, certs etc directories in the /persistent folder. Messy but not impacting.

As for the docker-compose file, this is what I have been testing with…

version: '2.4'
services:
  ziti-edge-router:
    #    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    image: test:latest
    environment:
      #- ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_HOSTNAME}"
      #- ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_HOSTNAME}"
      - ZITI_EDGE_ROUTER_HOSTNAME=ziti-edge-router
      - ZITI_EDGE_ROUTER_ROLES=public
      - ZITI_EDGE_ROUTER_PORT=443
    ports:
      - "443:3022"
    networks:
      - zitiblue
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-extrouter.sh edge"
networks:
  zitiblue:

volumes:
  ziti-fs:

The final step in the puzzle, is as you said, you need to copy the ziti.env file from the controller across to the persistent folder of the router to get all the environment variables defined. Not nice. Once the router is enrolled, then the ziti.env should be edited to remove the password variables.

So, to recap, once the modified container is built, the process would be as follows to get this working:

  1. Change the command in the docker-compose.yml to point to the run-extrouter.sh script
  2. Copy the ziti.env file in from the controller into the /persistent folder
  3. Define the ZITI_EDGE_ROUTER_ROLES, ZITI_EDGE_ROUTER_HOSTNAME environment variables
  4. Start the container

As a note, if you want to change the router name from the default, say call it my-router, you need to modify the ziti.env file and set the ZITI_EDGE_ROUTER_RAWNAME and then set the ZITI_EDGE_ROUTER_HOSTNAME through environment variables.

At least that is my experience.

FYI - Have submitted a PR to configure this within the docker environment: Adding ability to run external router by gooseleggs · Pull Request #966 · openziti/ziti (github.com)

1 Like

Thought to confirm my understanding… does this mean… that it allows you to create an external edge router for a docker config. so that the services running in the docker environment can be accessed remotely via an remote ziti cloud?

The docker quickstart puts everything on one host. Similar (most likely) to the normal client quickstart. What you cannot do using the docker container is stand up another router on another host somewhere (using docker) without a lot of manual steps of copying the ziti.env file, modifying it, and performing a number of manual steps like the standard install. The PR provided enables this to be accomplished through environment variable configuration of the docker-compose file. Basically, as per previous posts in this thread, we add a specific script for this scenario into the quickstart docker image meaning no copying of ziti.env around, editing it, or typing manual commands.

Well - it is my take on how it could be done.

1 Like