Running a second edge router on a different device

ajaykumaar · October 22, 2024, 9:23pm

Hi,

My objective is to have a controller/router on one device connected a network and a router on another device connected to a different network and to stream data between those two devices via ziti overlay network.

As a first step, I want to do this within the same network.

I saw a few threads on this topic but I'm wondering if there's a revised or simpler way to do this.

I have a controller and an edge router installed on an AMD platform (amd-1). I want to install a second router on another AMD platform (amd-2). I'm trying to follow the instructions in the documentation and the discourse but I keep getting errors along the way.

Steps i followed:

Start Controller and Router and ziti edge login on the amd-1 platform
Created an edge router on the amd-1 using:

ziti edge create edge-router "edge-router-2" --jwt-output-file edge-router-2.jwt --tunneler-enabled

Moved the generated jwt token to the amd-2 platform
Installed the openziti-router using the command

curl -sS https://get.openziti.io/install.bash | sudo bash -s openziti-router

Created the config file interactively, using the command: and made changes to the config.yml file

sudo /opt/openziti/etc/router/bootstrap.bash

At this point, I'm not able to use the ziti-router command on the terminal. Also, if i start the router service and check the status i get the below error:

I'd really appreciate any guidance or help on this. Thanks in advance.

-Ajay

TheLumberjack · October 22, 2024, 9:27pm

Hi @ajaykumaar, welcome to the community and to OpenZiti!

The first thing I'd tell you to do is to run the new, ziti ops verify-traffic command in the ziti cli to make sure you have everything setup properly.

Looks like you're using the linux package - so I'd expect you are good to go there but let's just make sure.

run it and enter the password (change the params accordingly):

ziti ops verify-traffic --host localhost --port 1280 --username admin

My guess is that it'll succeed on the local machine

Next, run that same command from the other machine. My guess is that it'll fail from the other machine.

Can we start with that?

ajaykumaar · October 23, 2024, 8:33pm

Hi @TheLumberjack ,

Thank you for your prompt response! While trying to run the ziti ops command, I realized I was using an older version of openziti so I removed the exiting installation and reinstalled the v1.1.15 version in both the machines.

As you've mentioned, on the local machine (amd-1), I'm able to run the ziti ops and get a clean output.

And I tried the same on the other machine (amd-2) and it failed

Also, I'm not able to do "ziti edge login" from the amd-2 machine. Here's the env variables for address and ports in amd-1 local machine
"""
ubuntu@ubuntu-ATOPNUC-MA90:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu-ATOPNUC-MA90
#0.0.0.0 ubuntu-ATOPNUC-MA90

#The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ubuntu@ubuntu-ATOPNUC-MA90:~$ echo ${ZITI_CTRL_ADVERTISED_ADDRESS}
ubuntu-ATOPNUC-MA90
ubuntu@ubuntu-ATOPNUC-MA90:~$ echo ${ZITI_CTRL_ADVERTISED_PORT}
6262
ubuntu@ubuntu-ATOPNUC-MA90:~$
ubuntu@ubuntu-ATOPNUC-MA90:~$ echo ${ZITI_ROUTER_PORT}
3022
ubuntu@ubuntu-ATOPNUC-MA90:~$
"""

Thank you,
Ajay

TheLumberjack · October 23, 2024, 8:36pm

Yes, that's what I guessed would happen. How did you start the first instance? I expect the PKI it's valid for is not for IP 192.168.1.83 (or for a matching hostname).

You'll want to reinstall the PKI but it'd be easiest to just dump the whole network and start over.

Make sure when you are bootstrapping you use a FQDN -- something that's routable from any/every client you want to use.

That make sense? If not, ask for clarity and I'll try to clear it up

EDIT: you might be able to get away with using your hosts file, but it'll be easier if whatever you use is routable by both/all machines... hopefully that's clear enough?

ajaykumaar · October 23, 2024, 9:55pm

Yes please, could you explain the "bootstrapping with FQDN" part?

To start the first instance (amd-1), I installed using the ExpressInstall command and the output said it's creating certificates for 127.0.0.1
"""
Creating server cert from ca: ubuntu-ATOPNUC-MA90-intermediate for localhost,ubuntu-ATOPNUC-MA90 / 127.0.0.1
Using CA name: ubuntu-ATOPNUC-MA90-intermediate
Success
Creating client cert from ca: ubuntu-ATOPNUC-MA90-intermediate for localhost,ubuntu-ATOPNUC-MA90
Using CA name: ubuntu-ATOPNUC-MA90-intermediate
Success

Creating server cert from ca: ubuntu-ATOPNUC-MA90-edge-controller-intermediate for localhost,ubuntu-ATOPNUC-MA90 / 127.0.0.1
Using CA name: ubuntu-ATOPNUC-MA90-edge-controller-intermediate
Success
Creating client cert from ca: ubuntu-ATOPNUC-MA90-edge-controller-intermediate for localhost,ubuntu-ATOPNUC-MA90
Using CA name: ubuntu-ATOPNUC-MA90-edge-controller-intermediate
Success

"""

I'm entirely new to this and I really appreciate your help.

Thank you,
Ajay

TheLumberjack · October 23, 2024, 10:34pm

Ok. Specifically, you want to make sure the ZITI_CTRL_ADVERTISED_ADDRESS is addressable from both/any computers you want to participate in the network.

I see from before you have this set but that is a hostname, not a FQDN:

ZITI_CTRL_ADVERTISED_ADDRESS=ubuntu-ATOPNUC-MA90

I expect this is not resolvable from your other host. When you install the controller, you need to make sure the address you pick is available to both machines. I would correct that and start over.

ajaykumaar · October 24, 2024, 6:21pm

Thanks for the response. So, I tried to create a controller from the "Deploy Controller" page and I was able to get the controller up and running with the correct ip address

ubuntu@ubuntu-ATOPNUC-MA90:~$ echo ${ZITI_CTRL_ADVERTISED_ADDRESS}
192.168.1.83

ubuntu@ubuntu-ATOPNUC-MA90:~$ sudo systemctl status ziti-controller.service
● ziti-controller.service - OpenZiti Controller
     Loaded: loaded (/lib/systemd/system/ziti-controller.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/ziti-controller.service.d
             └─override.conf
     Active: active (running) since Thu 2024-10-24 13:57:31 EDT; 12s ago
    Process: 12621 ExecStartPre=/opt/openziti/etc/controller/entrypoint.bash check config.yml (code=exited, status=0/SUCCESS)
   Main PID: 12642 (ziti)
      Tasks: 7 (limit: 8776)
     Memory: 54.6M
        CPU: 1.795s
     CGroup: /system.slice/ziti-controller.service
             └─12642 /opt/openziti/bin/ziti controller run config.yml -

But, when I tried to ziti edge login, I get this

ubuntu@ubuntu-ATOPNUC-MA90:~$ ziti edge login 192.168.1.83:1280 -u admin -p admin
Cert #0 in the chain doesn't match
WARNING: server supplied certificate authority doesn't match cached certs at /home/ubuntu/.config/ziti/certs/192.168.1.83
Replace cached certs [Y/N]: n
RESTY 2024/10/24 13:57:56 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, ::1, not 192.168.1.83, Attempt 1
RESTY 2024/10/24 13:57:56 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, ::1, not 192.168.1.83, Attempt 2

Looks like the certificates are still being generated for 127.0.0.1

Please let me know if I'm missing something.

Thank you,
Ajay

ajaykumaar · October 24, 2024, 6:22pm

Update:

It looks like the SANs still points to the loopback IP and doesn't have 192.168.1.83. So, I tried generating the certificates manually...

Created new root CA certificate

sudo ziti pki create ca --pki-root . --ca-file root --ca-name "Ziti Root CA"

Intermediate certificate

sudo ziti pki create intermediate --pki-root . --ca-name root --intermediate-file intermediate --intermediate-name "Ziti Intermediate CA"

Server certificate with 192.1668.1.83 in SANs

sudo ziti pki create server \
  --pki-root . \
  --ca-name intermediate \
  --server-file server \
  --server-name "Ziti Controller Server" \
  --dns "localhost,ziti_controller" \
  --ip "127.0.0.1,192.168.1.83"

Client certificate

sudo ziti pki create client \
  --pki-root . \
  --ca-name intermediate \
  --client-file client \
  --client-name "Ziti Controller Client"

And when I verify the SANs, it looks good

root@ubuntu-ATOPNUC-MA90:/var/lib/private/ziti-controller/pki# sudo openssl x509 -in intermediate/certs/server.chain.pem -text -noout | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti_controller, IP Address:127.0.0.1, IP Address:192.168.1.83

But then when I use s_client to check the SANs, I'm getting this:

root@ubuntu-ATOPNUC-MA90:/var/lib/private/ziti-controller/pki# echo | openssl s_client -connect 192.168.1.83:1280 -servername ziti_controller 2>/dev/null | openssl x509 -text -noout | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti_controller, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1

TheLumberjack · October 24, 2024, 7:08pm

Oh I wouldn't recommend this at this point. That's a whole world of pain I don't think we want to get into just yet. Getting the PKI "just right" is a delicate procedure and exceptionally easy to get wrong...

You have used the linux packaging, I would recommend you uninstall everything and start over. My hunch is that files are being leftover which is why your PKI isn't changing?

Can you start over from a clean install? That ok?

qrkourier · October 24, 2024, 7:21pm

The Linux controller issues new client and server leaves at startup. You can disable this behavior by setting 'false'.

❯ grep ZITI_AUTO_RENEW_CERTS /opt/openziti/etc/controller/service.env
ZITI_AUTO_RENEW_CERTS='true'

There's not currently a way to inject additional IP or DNS SANs to the auto-renewed leaves, but you can certainly manage those leaves yourself if you disable auto-renewal.

ajaykumaar · October 24, 2024, 9:17pm

Yeah, sure.

So, I delete everything as per the instruction in the "Controller Deployment" page and reinstalled the Controller and got it up and running (Image below)

The environment variables are clean so far and I'm able to login using the DNS name "ziti_controller" but the SANs still has a different IP address

ubuntu@ubuntu-ATOPNUC-MA90:~$ ziti edge login ziti_controller:1280 -u admin -p admin
Untrusted certificate authority retrieved from server
Verified that server supplied certificates are trusted by server
Server supplied 2 certificates
Trust server provided certificate authority [Y/N]: y
Server certificate chain written to /home/ubuntu/.config/ziti/certs/ziti_controller
Token: a622bc89-855e-4dd3-83b2-fdea608bc7c3
Saving identity 'default' to /home/ubuntu/.config/ziti/ziti-cli.json

ubuntu@ubuntu-ATOPNUC-MA90:~$ sudo -s
root@ubuntu-ATOPNUC-MA90:/home/ubuntu#  sudo openssl x509 -in /var/lib/private/ziti-controller/pki/intermediate/certs/server.chain.pem -text -noout | grep -
A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti_controller, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1
'''

qrkourier · October 24, 2024, 9:35pm

You issued a server cert with the additional IP SAN 192.168.1.83, but it's not showing up in the running controller's SANs, right?

Assuming you disabled auto-renew and manually issued the new cert and placed the server chain file in the same location, then the running controller will load the new cert on next startup. You will need to run sudo systemctl restart ziti-controller.service after you overwrite the cert chain file.

The default location is: /var/lib/private/ziti-controller/pki/intermediate/certs/server.chain.pem

You can issue a new server cert and place the chain in the default location with this command.

sudo ziti pki create server \
  --pki-root /var/lib/private/ziti-controller/pki \
  --ca-name intermediate \
  --server-file server \
  --server-name "Ziti Controller Server" \
  --dns "localhost,ziti_controller" \
  --ip "127.0.0.1,192.168.1.83" \
--allow-overwrite

ajaykumaar · October 24, 2024, 10:06pm

Yes, that worked!!

ubuntu@ubuntu-ATOPNUC-MA90:~$ sudo openssl x509 -in /var/lib/private/ziti-controller/pki/intermediate/certs/server.chain.pem -text -noout | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti_controller, IP Address:127.0.0.1, IP Address:192.168.1.83
ubuntu@ubuntu-ATOPNUC-MA90:~$
ubuntu@ubuntu-ATOPNUC-MA90:~$ echo | openssl s_client -connect 192.168.1.83:1280 -servername ziti_controller 2>/dev/null | openssl x509 -text -noout | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:ziti_controller, IP Address:127.0.0.1, IP Address:192.168.1.83

Thank you!!

Next up, I'm going to try installing the edge routers (er-1 on the local machine and er-2 on a different machine) and try to stream data through them.

Thank you,
Ajay

EDIT:
Just a quick question- after getting the controller running, I was trying to login with ziti edge and I'm able to login in using any of these commands below

ziti edge login ziti_controller:1280 -u admin -p admin
ziti edge login -u admin -p admin
ziti edge login 127.0.0.1:1280 -u admin -p admin

But, I'm not able to do so using the IP address,

ubuntu@ubuntu-ATOPNUC-MA90:~$ ziti edge login 192.168.1.83:1280 -u admin -p admin
Cert #0 in the chain doesn't match
WARNING: server supplied certificate authority doesn't match cached certs at /home/ubuntu/.config/ziti/certs/192.168.1.83
Replace cached certs [Y/N]: n
RESTY 2024/10/24 17:58:24 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate signed by unknown authority, Attempt 1
RESTY 2024/10/24 17:58:25 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate signed by unknown authority, Attempt 2
RESTY 2024/10/24 17:58:25 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate signed by unknown authority, Attempt 3
RESTY 2024/10/24 17:58:25 ERROR Get "https://192.168.1.83:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate signed by unknown authority, Attempt 4

Now that the SANs has both IP and the DNS name, I should be able to login in using the IP, right?

qrkourier · October 25, 2024, 8:39am

Yes. You can login with any SAN. You may have reissued the root from a new private key which would invalidate the cached root trust. Delete the cache location mentioned in the output or answer yes at the prompt to replace the cache.

ajaykumaar · October 25, 2024, 11:21pm

Hi,

Thanks for the guidance. I installed the edge router on the local machine, defined the necessary polices and the 'verify-traffic' runs clean.

Then, I installed another edge router on a different device in the same local network. I'm able to verify-traffic but when i check the status, I'm getting some errors and the edge-router shows offline when I print the edge-routers list

Is this similar to the controller certificate issue wherein I should generate a SANs certificate for DNS: ziti_controller and IP: 192.168.1.83 or Am I missing something else here?

Thank you,
Ajay

ajaykumaar · October 28, 2024, 6:28pm

I tried generating certificates and having the controller sign them but I'm still not able to fix it; maybe I'm not doing it right. Is that the only way to get router fabric up and running?

I'd really appreciate insights on this.

Thank you,
Ajay

qrkourier · October 28, 2024, 9:23pm

I believe you're troubleshooting edge-router-2 status online=false.

This status means the router failed to connect to the Ziti controller. The reason could be that the router hasn't been enrolled, started, or has an invalid configuration that caused it to quit before it connected.

What is the output from that router? It will log the attempt to connect to the Ziti controller at the address you configured in the config YAML file like this:

ctrl:
    endpoint: tls:ziti.example.com:443

Link to router configuration reference

ajaykumaar · October 30, 2024, 3:57pm

Hi @qrkourier
Yes, I'm troubleshooting edge-router-1 status online=false

After starting the router using sudo systemctl enable --now ziti-router.service, this is the output log

(base) root@ajay-ubuntu:/# sudo journalctl -u ziti-router.service -f
-- Logs begin at Sun 2023-06-04 10:40:32 EDT. --
Oct 30 11:49:05 ajay-ubuntu ziti[9370]: {"endpoint":"tls:192.168.1.83:1280","error":"error connecting ctrl (tls: failed to verify certificate: x509: certificate signed by unknown authority)","file":"github.com/openziti/ziti/router/env/ctrls.go:129","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2","level":"error","msg":"unable to connect controller","time":"2024-10-30T11:49:05.735Z"}
Oct 30 11:49:05 ajay-ubuntu ziti[9370]: {"endpoint":"tls:192.168.1.83:1280","error":"error connecting ctrl (tls: failed to verify certificate: x509: certificate signed by unknown authority)","file":"github.com/openziti/ziti/router/env/ctrls.go:129","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2","level":"error","msg":"unable to connect controller","time":"2024-10-30T11:49:05.830Z"}
Oct 30 11:49:05 ajay-ubuntu ziti[9370]: {"endpoint":"tls:192.168.1.83:1280","error":"error connecting ctrl (tls: failed to verify certificate: x509: certificate signed by unknown authority)","file":"github.com/openziti/ziti/router/env/ctrls.go:129","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2","level":"error","msg":"unable to connect controller","time":"2024-10-30T11:49:05.953Z"}

And this is the address I have in the config.yml

ctrl:
  endpoint:             tls:192.168.1.83:1280

I am able to ziti login using this IP.

ajay@ajay-ubuntu:~/ziti_codes$ ziti edge login 192.168.1.83:1280
Using username: admin from identity 'default' in config file: /home/ajay/.config/ziti/ziti-cli.json
Enter password:
Token: 86ccdad7-543c-47d4-b34e-f7ea5b3438ce
Saving identity 'default' to /home/ajay/.config/ziti/ziti-cli.json
ajay@ajay-ubuntu:~/ziti_codes$

So, I'm not sure what's causing the "error connecting to ctrl".

Thank you,
Ajay

qrkourier · October 30, 2024, 5:11pm

I think you may have recreated the controller and the offline router was enrolled with the old controller.

Thank you for pasting the log messages as text. This was the error:

error connecting ctrl (tls: failed to verify certificate: x509: certificate signed by unknown authority)

The router is offline because the controller listening at 192.168.1.83:1280 presents an unknown identity to the router.

You can re-enroll the offline router (edge-router-1?) with the current controller.

ziti edge re-enroll edge-router "edge-router-1" -o edge-router-1.jwt

Then, reset the router's state.

sudo systemctl disable --now ziti-router.service
sudo systemctl reset-failed ziti-router.service
sudo systemctl clean --what=state ziti-router.service

And re-generate the router's configuration. You may wish to double-check your input values in /opt/openziti/etc/router/bootstrap.env before you re-run /opt/openziti/etc/router/bootstrap.bash.

ajaykumaar · October 31, 2024, 2:53pm

Hi,

Thanks for the response. I tried re-enrolling the offline router

ajay@ajay-ubuntu:~/ziti_codes$ ziti edge login 192.168.1.83:1280
Using username: admin from identity 'default' in config file: /home/ajay/.config/ziti/ziti-cli.json
Enter password:
Token: 3e558afc-c185-406b-b6eb-74d81a0f1b85
Saving identity 'default' to /home/ajay/.config/ziti/ziti-cli.json
ajay@ajay-ubuntu:~/ziti_codes$
ajay@ajay-ubuntu:~/ziti_codes$
ajay@ajay-ubuntu:~/ziti_codes$ ziti edge re-enroll edge-router "edge-router-2" -o edge-router-2.jwt
re-enroll edge-router with id Z7.Z9.pswh: OK
Enrollment expires at 2024-10-30T22:35:22.131Z

I edited the bootstrap.env and ran the script but the router is still offline

ajay@ajay-ubuntu:~/ziti_codes$ sudo /opt/openziti/etc/router/bootstrap.bash
INFO: bootstrap completed successfully and will not run again. Adjust /var/lib/private/ziti-router/config.yml to suit.
ajay@ajay-ubuntu:~/ziti_codes$
ajay@ajay-ubuntu:~/ziti_codes$
ajay@ajay-ubuntu:~/ziti_codes$ ziti edge list edge-routers
╭────────────┬───────────────┬────────┬───────────────┬──────┬────────────╮
│ ID         │ NAME          │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼───────────────┼────────┼───────────────┼──────┼────────────┤
│ QHcUxCYpST │ edge-router-1 │ true   │ true          │    0 │            │
│ Z7.Z9.pswh │ edge-router-2 │ false  │ true          │    0 │            │
╰────────────┴───────────────┴────────┴───────────────┴──────┴────────────╯
results: 1-2 of 2

Thank you,
Ajay

Topic		Replies	Views
Setup new edge router on a separate host Ziti Overlay	11	633	January 17, 2023
Installing an edge router on a different compute Ziti Overlay	1	226	June 12, 2022
Creating a second public edge router Ziti Overlay	31	1407	June 18, 2024
New install help - ziti edge router is not available	5	34	August 19, 2024
Creating an edge router on a server different to the controller Building/Development	8	627	August 7, 2022

Running a second edge router on a different device

Related topics