I’ll start with OCI and make a video from start to finish. Gimme a bit. I can see the cert output from openssl still has this in it:
X509v3 Subject Alternative Name:
DNS:144.24.195.21, DNS:10.0.0.235
If you want me to use your controller - cool. I’m happy to do that 
i’ll DM you now
Ok. I got to the bottom of the issue. It comes down to a few things but mainly it’s because the command I provided to @markamind on his other post was ever so slightly incorrect and that caused a lot of confusion! 
I’ll update that post after this one.
@arslane gave me access to his OCI instance saving me a bit of time making an OCI machine. I wanted to use it anyway, to save you the time and so you’ll have your own instance that’s configured correctly as reference. Here’s what I needed to do.
There are four main ports needed to add a second router to a quickstart-enabled instance. The first two are always opened as part of the quickstart, the second two are only opened when you add a second router. These are those ports:
First thing I did was to check that all these “server” ports all returned a certificate with DNS names properly setup. Port 6262 was still returning the IP in the DNS field. That was when I realized that the instructions I gave to @markamind were incomplete.
The quickstart comes with three PKI’s setup for the controller/edge controller as it’s meant to be educational. There’s a PKI for the overall network - the ‘controller’ (port 6262). One for the edge controller which is not quite the same as the ‘controller’. This is the PKI for the APIs (port 8441). Both certs need to be fixed. To do that I sourced the .env file that comes with the quickstart:
. $HOME/.ziti/quickstart/$(hostname)/$(hostname).env
Then I setup the variables but notice I did the DNS-related ones differently. Also note that I have two different files called out at the bottom of this code block… one for each cert we’re about to make. It’s vital that you do not set the DNS related fields to an IP address. Here you can see I made up one, and used the hostname for the other:
now="$(date +'%Y-%m-%d_%H%M%S')"
EDGE_CONTROLLER_EXTERNAL_DNS_NAME=my.private.ziti.controller
EDGE_CONTROLLER_PRIVATE_DNS_NAME=$(hostname)
EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS=144.24.195.21
EDGE_CONTROLLER_PRIVATE_IP_ADDRESS=10.0.0.235
pki_allow_list_dns="${EDGE_CONTROLLER_EXTERNAL_DNS_NAME},${EDGE_CONTROLLER_PRIVATE_DNS_NAME},localhost,$(hostname)"
pki_allow_list_ip="127.0.0.1,${EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS},${EDGE_CONTROLLER_PRIVATE_IP_ADDRESS}"
new_ctrl_cert_name="${ZITI_CONTROLLER_HOSTNAME}-${now}"
new_edge_ctrl_cert_name="${ZITI_EDGE_CONTROLLER_HOSTNAME}-${now}"
generated a new server cert for the controller by running:
"${ZITI_BIN_DIR}/ziti" pki create server \
--pki-root="${ZITI_PKI_OS_SPECIFIC}" \
--ca-name ${ZITI_CONTROLLER_INTERMEDIATE_NAME} \
--key-file "${ZITI_CONTROLLER_HOSTNAME}-server" \
--server-file "${new_ctrl_cert_name}-server" \
--dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
--server-name "${new_ctrl_cert_name} server certificate"
generated a new server cert for the edge controller by running:
"${ZITI_BIN_DIR}/ziti" pki create server \
--pki-root="${ZITI_PKI_OS_SPECIFIC}" \
--ca-name ${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME} \
--key-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server" \
--server-file "${new_edge_ctrl_cert_name}-server" \
--dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
--server-name "${new_edge_ctrl_cert_name} server certificate"
Finally, we can find and output these two files:
cat <<HERE
CONTROLLER server cert use:
$(find $ZITI_HOME -name "*${new_ctrl_cert_name}*chain.pem")
EDGE CONTROLLER server cert use:
$(find $ZITI_HOME -name "*${new_edge_ctrl_cert_name}*chain.pem")
HERE
which looked like this:
CONTROLLER server cert use:
/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-2022-08-07_190024-server.chain.pem
EDGE CONTROLLER server cert use:
/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/certs/144.24.195.21-2022-08-07_190024-server.chain.pem
With those files in hand, now I could edit the controller’s config file:
vi $ZITI_HOME/$(hostname).yaml
The first thing I did was edit the identity.server_cert near the top of the file and replaced the server_cert with the NEW server cert. Then I found the edge controller server_cert location in the web.identity.server_cert section (near the bottom of the file) and I updated that value.
Finally I restarted the controller:
sudo systemctl restart ziti-controller
I wanted to make sure @arslane would be in good shape, so I make a “clint-aws” edge router:
ziti edge create edge-router clint-aws -o clint-aws.jwt -t
I scp’ed the jwt from @arslane’s instance, and scp’ed it to my AWS router. Then I cd’ed to that $ZITI_HOME, copied my existing config file, changed the necessary settings, enrolled it, and tried to start that router. When I did that the first time, my edge router in AWS reported “link dialing failed”:
certificate is valid for 127.0.0.1, not 144.24.195.21] linkId=[4xwLnaMbpcofMRpOsYnaR6]} link dialing failed
There seems to be another bug with the quickstart though because the edge-router’s config had an IP address in the SANS section. It looked like this:
sans:
dns:
- 144.24.195.21
- localhost
ip:
- "127.0.0.1"
I updated that section to use the hostname in the dns section and to use the external ip in the ip section. It now looks like this:
sans:
dns:
- "instance-20220807-1808"
- "localhost"
ip:
- "127.0.0.1"
- "144.24.195.21"
The problem is that the certificates for routers are created during the enrollment process. That meant I had to delete the existing edge-router, and recreate it. Using the ziti-cli-functions.sh as a guide, I did that using the following commands:
ZITI_EDGE_ROUTER_RAWNAME="${ZITI_NETWORK}-edge-router"
"${ZITI_BIN_DIR-}/ziti" edge delete edge-router "${ZITI_EDGE_ROUTER_RAWNAME}"
"${ZITI_BIN_DIR-}/ziti" edge create edge-router "${ZITI_EDGE_ROUTER_RAWNAME}" -o "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt" -t -a "public"
"${ZITI_BIN_DIR-}/ziti-router" enroll "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.yaml" --jwt "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt"
sudo systemctl start ziti-router
I verified the ‘public’ edge router started (Arslane’s edge router, it did start) and then went back to my AWS router and started that router. This time it connected to Arslane’s public edge-router. I could verify that by looking at the links formed:
ziti fabric list links
╭────────────────────────┬───────────┬────────────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE │ STATUS │ FULL COST │
├────────────────────────┼───────────┼────────────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 7hx4qaAE1UlL4iPIH51Phg │ clint-aws │ instance-20220807-1808-edge-router │ 1 │ 65000.0ms │ 65000.0ms │ Connected │ up │ 130001 │
╰────────────────────────┴───────────┴────────────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-1 of 1
That was a lot more effort than I thought, and would be VERY hard for someone not as familiar with X509/the PKI setup of OpenZiti to figure out!!!
Thanks @arslane for letting me use your OCI instance. I look forward to your follow-up.
Here’s a video taking you from nothing in OCI through the express install, through the addition of a second, private edge router. It’s basically the same steps as above, just cleaner, and narrated…
Thanks @TheLumberjack … this is something that I have been trying to do for some time now… and will be working on this today… so this is great timing.
Something I also noticed is that you created the jwt file on the controller… and then copied to the remote router.
My understanding is that this is because you have only allowed local host access to the controller.
Nah it’s just because I’m lazy and when I use a quickstart-installed system I’ll just ssh to that machine because it’ll have zitiLogin on it. I absolutely could have ssh’ed onto the edge-router I was installing, used ziti login to login to my controller from there, and created the .jwt on the router. Then I wouldn’t have to scp the .jwt around.
Is there anyway you can restrict the Ziti Login from a remote machine? Curious to know.
Yes, we covered that on a separate discourse post about a month or two back here Making ZAC dark - #6 by dovholuknf
You need to “split” the api. That’s done in the controller. The relevant changelog entry talks about it too (also in that thread) Making ZAC dark - #3 by dovholuknf
And for help - here’s what my current ‘web’ config looks like on my aws instance where i have the fabric and management apis split from the edge-client…
web:
- name: dark-api
bindPoints:
- interface: 0.0.0.0:18441
address: localhost:18441
identity:
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate.cert"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.key"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server-2022-06-10-3915.chain.pem"
_server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.chain.pem"
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-client.cert"
apis:
- binding: edge-management
options: { }
- binding: fabric
options: { }
- name: client-management
bindPoints:
- interface: 0.0.0.0:8441
address: ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8441
identity:
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate.cert"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.key"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server-2022-06-10-3915.chain.pem"
_server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.chain.pem"
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-client.cert"
options:
idleTimeout: 5000ms #http timeouts, new
readTimeout: 5000ms
writeTimeout: 100000ms
minTLSVersion: TLS1.2
maxTLSVersion: TLS1.3
apis:
- binding: edge-client
options: { }
i do NOT have port 18441 exposed through the AWS ACL (security group) so you can only access the api from the local machine…
Oh - and if you DO do that, you’ll need to fix your zitiLogin alias too since it’ll only work on “localhost” now… instead of:
alias zitiLogin='ziti edge login "${ZITI_EDGE_CTRL_ADVERTISED}"/edge/management/v1/ -u "${ZITI_USER-}" -p "${ZITI_PWD}" -c "${ZITI_PKI}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/certs/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}.cert"'
you would use:
alias zitiLogin='ziti edge login localhost:18441/edge/management/v1/ -u "${ZITI_USER-}" -p "${ZITI_PWD}" -c "${ZITI_PKI}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/certs/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}.cert"'
PS… I know what causes the problem with scp for OCI. its the command you added into the bash profile. this apparently mucks it up… and errors… once you comment this out… it works as normal
Interesting! Yes you’re right. Apparently the echo during the sourcing causes the issue. Now I want to understand why Ubuntu doesn’t seem to have that particular problem. 
Thanks for pointing that out!
The Oracle Linux certainly has its points of difference
Thanks for sharing these specific details… I was not 100% sure about how to set this up…
I also remembered that I did not explain what i did to fix the command and how the zitiLogin was breaking. In my original post to @markamind I had neglected to use one special field which I thought I transcribed. All the certs, the client certs and the server certs need to use the same key. In the original command I forgot to pass the --key-file parameter. I fixed that error above with the highlighted line:
This was the missing step. During the process I also discovered that I only gave him one of two needed commands (the controller pki) and forgot about the edge pki.
Slowly working through all of this… I have most things worked out now… and are working on rebuilding the server and router certificates
This has been a really valuable learning exercise.
I am not sure I follow this step. Is this one command? What does HERE mean?
Is this step making the chain.pem file foe each of the certificates… if so… how does it make the file name
My bash skills still needs more work
just copy and paste the whole block and it’ll execute. Alternatively, just execute these commands:
# find the CONTROLLER server cert use:
$(find $ZITI_HOME -name "*${new_ctrl_cert_name}*chain.pem")
# find EDGE CONTROLLER server cert use:
$(find $ZITI_HOME -name "*${new_edge_ctrl_cert_name}*chain.pem")
Then if you’re interested to learn more, lookup and read about heredocs . It was an attempt to make it easier than harder - which seems to have failed!
No… no failure… and in fact… very helpful… as it helps me learn more.
What is the meaning of the _server_cert identity property on a web bind point?
it’s an easy way to backup an entry in yaml or in json… just prepend it with an underscore so that the parser just kinda ignores it.
i often prepend the underscore like that so it’s easy to ‘undo’ if i need to.