What is the easiest way to create an HA setup with 3 controllers

8

Ok Here's what I did. The most complex part of clustered controllers is understanding the PKI needs to be generated from the same root CA. To do that you either need to generate the PKI on the same machine and transfer intermediates later on, or as in this example, you need to transfer the root cert and root key to each node. (after bootstrapping, you should remove the root key and offline it and keep it very safe of course).

I tried to parameterize it to keep it easy on you as well. I have a wildcard domain setup to *.zrok.clint.demo.openziti.org too but I think this is hopefully enough to get you going... Have a look and see if you have any questoins.

Initial Controller - ctrl1

export TRUST_DOMAIN="zrok.clint.demo.openziti.org"
export ZITI_PWD="replace.this"
export ZITI_INST="ctrl1"
export ZITI_CTRL_PORT="6400"
export ZITI_ROUTER_PORT="6401"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:${ZITI_CTRL_PORT}"
sudo chown ziti:ziti /sharedfs/
ziti edge quickstart ha \
    --instance-id="ctrl1" \
    --ctrl-port="${ZITI_CTRL_PORT}" \
    --router-port="${ZITI_ROUTER_PORT}" \
    --home="/sharedfs/ziti" \
    --ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --trust-domain="${TRUST_DOMAIN}" \
    --password $ZITI_PWD

The Other Two Controllers

Notice here the command differs slightly from before. Also notice that i used "cp" to transfer the root key/cert/index.txt file. That's so the quickstart command can generate the necessary PKI for you when it runs...

Ctrl2

export TRUST_DOMAIN="zrok.clint.demo.openziti.org"
export ZITI_PWD="replace.this"
export ZITI_INST="ctrl2"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:6400"
export ZITI_CTRL_PORT="6500"
export ZITI_ROUTER_PORT="6501"

mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/keys"
mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/certs"
cp /sharedfs/ziti/pki/root-ca/keys/root-ca.key /tmp/${ZITI_INST}/pki/root-ca/keys/
cp /sharedfs/ziti/pki/root-ca/certs/root-ca.cert /tmp/${ZITI_INST}/pki/root-ca/certs/
cp /sharedfs/ziti/pki/root-ca/index.txt /tmp/${ZITI_INST}/pki/root-ca/index.txt

ziti edge quickstart join \
    --instance-id "${ZITI_INST}" \
    --ctrl-port "${ZITI_CTRL_PORT}" \
    --router-port "${ZITI_ROUTER_PORT}" \
    --home "/tmp/${ZITI_INST}" \
    --ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --trust-domain="${TRUST_DOMAIN}" \
    --cluster-member "${ZITI_INITIAL_CTRL}" \
    --password $ZITI_PWD

Ctrl3

export TRUST_DOMAIN="zrok.clint.demo.openziti.org"
export ZITI_PWD="replace.this"
export ZITI_INST="ctrl3"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:6400"
export ZITI_CTRL_PORT="6600"
export ZITI_ROUTER_PORT="6601"

mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/keys"
mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/certs"
cp /sharedfs/ziti/pki/root-ca/keys/root-ca.key /tmp/${ZITI_INST}/pki/root-ca/keys/
cp /sharedfs/ziti/pki/root-ca/certs/root-ca.cert /tmp/${ZITI_INST}/pki/root-ca/certs/
cp /sharedfs/ziti/pki/root-ca/index.txt /tmp/${ZITI_INST}/pki/root-ca/index.txt

ziti edge quickstart join \
    --instance-id "${ZITI_INST}" \
    --ctrl-port "${ZITI_CTRL_PORT}" \
    --router-port "${ZITI_ROUTER_PORT}" \
    --home "/tmp/${ZITI_INST}" \
    --ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --trust-domain="${TRUST_DOMAIN}" \
    --cluster-member "${ZITI_INITIAL_CTRL}" \
    --password $ZITI_PWD

After Running These Commands Overview

These commands will end up making three controllers and three rotuers using the current quickstart command. (future readers, the ha subcommand should be going away, just run quickstart without ha)

If you shut down a controller, the config files and pki will all remain for you to inspect if you wish. Also you can choose to run the controller and router separately at that point too. For example you could run something like ziti controller run /tmp/ctrl3/ctrl3/ctrl.yaml to run the third controller without a router as I did below (rotuer 3 is offline):

ziti fabric list routers
╭────────────┬──────────────┬────────┬──────┬──────────────┬──────────┬───────────────────────┬────────────────────────────────────────────────╮
│ ID         │ NAME         │ ONLINE │ COST │ NO TRAVERSAL │ DISABLED │ VERSION               │ LISTENERS                                      │
├────────────┼──────────────┼────────┼──────┼──────────────┼──────────┼───────────────────────┼────────────────────────────────────────────────┤
│ UkAcDuu-WG │ router-ctrl2 │ true   │    0 │ false        │ false    │ v1.4.3 on linux/amd64 │ 1: tls:ctrl2.zrok.clint.demo.openziti.org:6501 │
│ cB-jbKK3IS │ router-ctrl3 │ false  │    0 │ false        │ false    │                       │                                                │
│ rjla7eevYs │ router-ctrl1 │ true   │    0 │ false        │ false    │ v1.4.3 on linux/amd64 │ 1: tls:ctrl1.zrok.clint.demo.openziti.org:6401 │
╰────────────┴──────────────┴────────┴──────┴──────────────┴──────────┴───────────────────────┴────────────────────────────────────────────────╯
results: 1-3 of

Ok, I'll stop here and let you digest this. I hope it helps and isn't overwhelming or not what you're looking for! :slight_smile:

1 Like