How to deploy and add a controller to an existing cluster

[Fabio72]

I have successfully migrate my quickstart/express install of controller to an HA one node cluster.
I followed this post

Move to HA set up from non HA

Hi @Fabio72 welcome to the community and to OpenZiti! that's great! Nice! From the quickstart -- no this won't be sufficient. I have been getting ready to comment about HOW one might actually accomplish this. MAKE SURE YOU BACKUP FIRST Here are my notes. Given what you've already accomplished, I expect you'll hit no issues here and realize you need all the variables set properly... MAKE SURE YOU BACKUP FIRST Did you backup??? You can just cp -r the entire quick…

to recreate server certificates including the missing SPIFFE ID.

Now I wish to add a second node.
Studying the file create-pki.sh in HA doc folder, I deduce the only common part between nodes id is the pki CA.
So I should deploy a new controller using the CA from the existing one.
Is it possible to bootstrap or express install a controller importing an existing CA?

My release is 1.2.2. Should I upgrade to 1.3.3 before proceeding?

Kind Regards

[TheLumberjack]

@Fabio72

So I should deploy a new controller using the CA from the existing one.

Yes. At this time, you will need to create an intermediate from the original server and then transfer the cert/key to the second controller. We are going to address this flow at some point in the future so there will be no need to transfer a private key, but at this time, that's what you'll need/want to do.

By far the easiest way to do this will be to ssh your root CA and root key to the target machine. If you want to use a quickstart, you should be able to but just beware you and I are breaking new ground here... I haven't run through this myself yet. (it's on my list todo, just isn't todone yet) Also Ken is starting on the work to do this under the 'deployments' umbrella so we are working to make all this easier in the future.

If it were me, I would start by running both controllers on the same machine. That will give you what you need and let you iterate and make sure things are working. Once it works on one machine, I would just pick up my configuratoin and scp it to the other machine and clean up the pki bits that aren't needed (the ones ctrl 1 use)

That make sense?

[Fabio72]

Yes, makes sense. So I need to generate the intermediate, server and client for second controller on my ctrl1.
Then I need to copy and modify the yaml configuration for ctrl2, with different ports.

Since the ctrl2 will use a different DNS name, I need to temporarly assign a second name to my ctrl1 host.

[TheLumberjack]

Yeah that seems about right. You could probably just use a hosts file entry on controller 1 as well. You could also just scp the full pki to controller 2 firstt, get the second controller working and then rm the bits too I suppose. Then you can skip the temporary DNS stuff...

I haven't really gone through the process myself, so forgive me for not having a definitive answer here. I look forward to your results though...

[Fabio72]

This is what I figured out:

export EXTERNAL_DNS=gziti.mydomain.my
export EXTERNAL_DNS_ctrl1=ziti.mydomain.my
export EXTERNAL_IP=x.x.x.x
# first node ID
#export SPIFFE_ID=oci
#second node ID
export SPIFFE_ID=gcp

export ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME="${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME/controller/controller2}"

export ZITI_PKI_CTRL_INTERMEDIATE_NAME="$EXTERNAL_DNS"-intermediate

ziti pki create intermediate \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_ROOTCA_NAME" \
	--intermediate-file "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--intermediate-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" 

ziti pki create intermediate \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_ROOTCA_NAME" \
	--intermediate-file "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--intermediate-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" 

ziti pki create intermediate \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--intermediate-file "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--intermediate-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" 
	
	
ziti pki create server \
	--spiffe-id  spiffe://$EXTERNAL_DNS_ctrl1/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file  $ZITI_PKI_CTRL_INTERMEDIATE_NAME \
	--expire-limit 1200 \
	--server-file controller2.2025.server \
	--server-name $SPIFFE_ID

ziti pki create client \
	--spiffe-id  spiffe://$EXTERNAL_DNS_ctrl1/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file gziti.mydomain.my-intermediate \
	--expire-limit 1200 \
	--client-file controller2.2025.client \
	--client-name $SPIFFE_ID

export NEW_CTRL_SERVER_CERT=$(realpath $(find . -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))
export NEW_CTRL_CLIENT_CERT=$(realpath $(find . -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))

ziti pki create server \
	--spiffe-id  spiffe://$EXTERNAL_DNS_ctrl1/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME \
	--expire-limit 1200 \
	--server-file controller2.2025.server \
	--server-name $SPIFFE_ID
ziti pki create client \
	--spiffe-id  spiffe://$EXTERNAL_DNS_ctrl1/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME \
	--expire-limit 1200 \
	--client-file controller2.2025.client \
	--client-name $SPIFFE_ID
	
export NEW_EDGE_SERVER_CERT=$(realpath $(find . -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))
export NEW_EDGE_CLIENT_CERT=$(realpath $(find . -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))


echo "Update your controller config file. "
echo "Assuming you ran a quickstart, you should have a separate PKI for CTRL/EDGE"
echo ""
echo "Modify the control plane identity block with:"
echo "server_cert: \"${NEW_CTRL_SERVER_CERT}\""
echo "cert:        \"${NEW_CTRL_CLIENT_CERT}\""
echo ""
echo "Modify the edge api identity block with:"
echo "server_cert: \"${NEW_EDGE_SERVER_CERT}\""
echo "cert:        \"${NEW_EDGE_CLIENT_CERT}\""

I run it on a second host with a cloned .ziti folder from 1st host.
all certificates where successfully created.

Now I'm stuck on another issue

[Fabio72]

I thing I got the gist of it.
I had much more struggling than necessary due to a firewall rule allowing only european IP addresses to connect to my home infrastructure. D'ho!
My first node (controller + router) is on oracle cloud in Europe so it was not a big deal.
My second node, though, is on google cloud in US, so connections and services were jerky.

Detailed steps:
All binaries version 1.3.3
First node installed as Quickstart Host Openziti Anywhere

1. recreate server/client certificates whith SPIFFE ID, following Clint's suggestion:

export ROOT_OF_TRUST=mydomain.my
export EXTERNAL_DNS=ziti.mydomain.my
export EXTERNAL_IP=123.123.123
export SPIFFE_ID=oci

# 1st node
# ctrl Plane
ziti pki create server \
	--spiffe-id spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file "${EXTERNAL_DNS}-server" \
	--server-file controller_$SPIFFE_ID.2025.server \
	--server-name $SPIFFE_ID
ziti pki create client \
	--spiffe-id  spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file "${EXTERNAL_DNS}-server" \
	--client-file controller_$SPIFFE_ID.2025.client \
	--client-name $SPIFFE_ID

NEW_CTRL_SERVER_CERT=$(realpath $(find .ziti -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))
NEW_CTRL_CLIENT_CERT=$(realpath $(find .ziti -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))
# edge
ziti pki create server \
	--spiffe-id  spiffe://$EXTERNAL_DNS/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file "${EXTERNAL_DNS}-server" \
	--server-file controller_$SPIFFE_ID.2025.server \
	--server-name $SPIFFE_ID
ziti pki create client \
	--spiffe-id  spiffe://$EXTERNAL_DNS/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file "${EXTERNAL_DNS}-server" \
	--client-file controller_$SPIFFE_ID.2025.client \
	--client-name $SPIFFE_ID

NEW_EDGE_SERVER_CERT=$(realpath $(find .ziti -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))
NEW_EDGE_CLIENT_CERT=$(realpath $(find .ziti -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))

echo "Update your controller config file. "
echo "Assuming you ran a quickstart, you should have a separate PKI for CTRL/EDGE"
echo ""
echo "Modify the control plane identity block with:"
echo "server_cert: \"${NEW_CTRL_SERVER_CERT}\""
echo "cert:        \"${NEW_CTRL_CLIENT_CERT}\""
echo ""
echo "Modify the edge api identity block with:"
echo "server_cert: \"${NEW_EDGE_SERVER_CERT}\""
echo "cert:        \"${NEW_EDGE_CLIENT_CERT}\""

2. replace certs as indicated
3. enable the raft section on yaml configuration and reboot the controller to get a running one node cluster.
4. copy the entire .ziti folder on second node
5. modify yaml file with correct advertiseaddress. Remove db stanza from configuration. Modify env file, only for the controller part. Do not change hostname part of folders.
6. remove any content of raft and db folders
7. create new intermediate, server and client certificates and update yaml accordly:

#2nd node
export ROOT_OF_TRUST=mydomain.my
export EXTERNAL_DNS=gziti.mydomain.my
export EXTERNAL_IP=321.321.321.321
export SPIFFE_ID=gcp
export CTRL1_HOSTNAME=hostname1

export ZITI_PKI_CTRL_INTERMEDIATE_NAME=gziti.mydomain.my-intermediate
export ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME=$CTRL1_HOSTNAME-edge-controller-$SPIFFE_ID-intermediate
#change env accordly

#ctrl plane
ziti pki create intermediate \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_ROOTCA_NAME" \
	--intermediate-file "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--intermediate-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" 
	
ziti pki create server \
	--spiffe-id spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file  $ZITI_PKI_CTRL_INTERMEDIATE_NAME \
	--server-file controller_$SPIFFE_ID.2025.server \
	--server-name $SPIFFE_ID
ziti pki create client \
	--spiffe-id  spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_INTERMEDIATE_NAME" \
	--key-file  $ZITI_PKI_CTRL_INTERMEDIATE_NAME \
	--client-file controller_$SPIFFE_ID.2025.client \
	--client-name $SPIFFE_ID

NEW_CTRL_SERVER_CERT=$(realpath $(find .ziti -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))
NEW_CTRL_CLIENT_CERT=$(realpath $(find .ziti -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))	
NEW_CTRL_KEY=$(realpath $(find .ziti -name "*2025*server*key" | grep $ZITI_PKI_CTRL_INTERMEDIATE_NAME))	

#edge
ziti pki create intermediate \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_ROOTCA_NAME" \
	--intermediate-file "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--intermediate-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" 	
	
ziti pki create server \
	--spiffe-id spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--dns "$EXTERNAL_DNS,$(hostname),localhost" \
	--ip "$EXTERNAL_IP,127.0.0.1" \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME \
	--server-file controller_$SPIFFE_ID.2025.server \
	--server-name $SPIFFE_ID
ziti pki create client \
	--spiffe-id spiffe://$ROOT_OF_TRUST/controller/$SPIFFE_ID \
	--pki-root "$ZITI_PKI" \
	--ca-name "$ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME" \
	--key-file $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME \
	--client-file controller_$SPIFFE_ID.2025.client \
	--client-name $SPIFFE_ID

NEW_EDGE_SERVER_CERT=$(realpath $(find .ziti -name "*2025*server*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))
NEW_EDGE_CLIENT_CERT=$(realpath $(find .ziti -name "*2025*client*chain*" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))
NEW_EDGE_KEY=$(realpath $(find .ziti -name "*2025*server*key" | grep $ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME))


echo "Update your controller config file. "
echo "Assuming you ran a quickstart, you should have a separate PKI for CTRL/EDGE"
echo ""
echo "Modify the control plane identity block with:"
echo "server_cert: \"${NEW_CTRL_SERVER_CERT}\""
echo "cert:        \"${NEW_CTRL_CLIENT_CERT}\""
echo "key:        \"${NEW_CTRL_KEY}\""
echo ""
echo "Modify the edge api identity block with:"
echo "server_cert: \"${NEW_EDGE_SERVER_CERT}\""
echo "cert:        \"${NEW_EDGE_CLIENT_CERT}\""
echo "key:        \"${NEW_EDGE_KEY}\""

8. configure systemd file, start the controller and check for errors. Controller should complain about missing admin profile.
9. add node to cluster with command:

ziti agent cluster -i oci --id gcp add tls:gziti.mydomain.my:8440

10. modify routers configuration file adding endpoints list

I had to restart apps and tunnels, but now everything seem fine.