Controller advertised address is ignoring port

Ziti Overlay
1

Hi - I'm using Ziti v1 and, it's great; but migrating to v2 is proving a challenge!

Our primary controller and router are on a single host, accessed via Nginx on port 443 with proxy destination selected by hostname.
We have found firewalls and VPNs block non-standard ports, and 443 is nearly always permitted; hence we want to use port 443.

  • controller.example.com:443 -> port 1280
  • router.example.com:443 -> port 3022

Here is our controller configuration

v: 3

cluster:
  dataDir:         "/var/lib/ziti-controller/raft"

identity:
  cert:        "pki/intermediate/certs/client.chain.pem"
  server_cert: "pki/intermediate/certs/server.chain.pem"
  key:         "pki/intermediate/keys/server.key"
  ca:          "pki/root/certs/root.cert"

ctrl:
  options:
    advertiseAddress: tls:controller.example.com:443
    # newListener: tls:controller.example.com:443
  listener: tls:0.0.0.0:1280

healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s

edge:
  api:
    sessionTimeout: 30m
    address: controller.example.com:443
  enrollment:
    signingCert:
      cert: pki/intermediate/certs/intermediate.cert
      key:  pki/intermediate/keys/intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m

web:
  - name: client
    bindPoints:
      - interface: 0.0.0.0:1280
        address: controller.example.com:443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: edge-oidc
        options: { }
      - binding: health-checks
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:8444
        address: controller.example.com:8444
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: edge-oidc
        options: { }
  - name: zac
    bindPoints:
      - interface: 0.0.0.0:18441
        address: controller.example.com:18441
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
      alt_server_certs:
        - server_cert: "/etc/letsencrypt/live/example.com/fullchain.pem"
          server_key:  "/etc/letsencrypt/live/example.com/privkey.pem"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: spa
        options:
          path: zac
          location: /opt/openziti/share/console
          indexFile: index.html

However....
The command line shows the controller is at port 1280, not port 443!

ubuntu@ip-N.N.N.N:~$ sudo ziti agent cluster list
╭────────────┬─────────────────────────────────┬───────┬────────┬─────────┬───────────┬───────────╮
│ ID         │ ADDRESS                         │ VOTER │ LEADER │ VERSION │ CONNECTED │ PREFERRED │
├────────────┼─────────────────────────────────┼───────┼────────┼─────────┼───────────┼───────────┤
│ controller │ tls:controller.example.com:1280 │ true  │ false  │ v2.0.4  │ true      │ false     │
╰────────────┴─────────────────────────────────┴───────┴────────┴─────────┴───────────┴───────────╯

This causes routers a problem.
The routers are configured to communicate with controller port 443.
First launch of a router works fine and it shows online in ZAC.
However, the endpoints file stores port 1280 (not 443).
When the router is restarted, it fails to communicate with the controller as it is trying to use port 1280 which is not a public port.

I'd say this looks like a simple bug, but the controller documentation makes me think otherwise.

listener - (required) is in the format of <protocol>:<interface>:<port> format. The value set here must be resolvable by routers and correspond the routers ctrl.endpoint configuration value.

I cannot put 443 into listener as the controller will try to listen on this port; it is already used by Nginx.
1280 is the only port number that makes sense, but it cannot be resolved by other routers.

TLDR
How to configure a controller to listen at one port with a different port advertised to networked components?
It is not clear how to write the controller configuration to achieve this goal.

2

FYI - this is related to What rewrites and loads endpoints.yml when ctrl.endpointsFile is commented out?
This post only covers the controller, the related post starts by using the router to diagnose incorrectly published port numbers.

3

I may have some progress.
I purged Ziti controller, shutdown Nginx, and reinstalled using port 443. Ziti Command line and ZAC worked fine.
Then I updated the controller config to listen on port 1280 and restarted Nginx and Ziti. Ziti Command line and ZAC worked fine. Significantly, the agent cluster list still points to 443 - Woot Woot

Looks highly likely plorenz was right - singleton rafts do not update correctly

Currently attempting to launch the router using this controller (first attempt failed to connect but that might be finger trouble); I'll let you know how it goes.

4

Hi @duncan.simey ,
Glad to hear you made some progress. If it's still not working, let me know. I've updated the issue: Sync a controller's raft advertise address with its config on startup · Issue #3588 · openziti/ziti · GitHub to reflect what I'm working on to fix this.

Paul

5

Thanks! The problem statement in #3588 covers the situation nicely!

Please could you elaborate on "the only recovery is to snapshot the database, clear raft, and re-bootstrap" - this sounds useful, but to be honest I don't think I could do this without some guidance.
I already tried deleting the raft DB and a new DB was created but Ziti did not come to life. I suspect I was fairly close but didn't do all the needed steps. For example snapshotting the main DB must be for it to be replaced at some point?
I think there may be circumstances where your steps would be more convenient than purging Ziti and installing at 443.

6

Hi @duncan.simey

Quick note, the fix is up for review: Reconcile the raft advertise address with the controller config on startup. Fixes #3588 by plorenz · Pull Request #4477 · openziti/ziti · GitHub

So to fix this before then, here are some instructions, assuming you're using v2.0.+

Note instructions were generated, then reviewed for accuracy and fixed up manually.

With a single controller, the fix is to rebuild the cluster from a copy of your data. It then gets bootstrapped fresh, and bootstrapping records the advertiseAddress from your config. Your identities, routers, services, policies and certificates all come across. When you deleted the raft directory before, you were most of the way there: the controller came up uninitialized and was waiting for a database to restore from, which is the restore-from-db step below.

These steps assume the Linux package layout from your posts: controller state in /var/lib/ziti-controller, cluster.dataDir set to /var/lib/ziti-controller/raft, and ctrl.options.advertiseAddress already set to tls:controller.example.com:443.

1. Snapshot the database while the controller is running

sudo ziti agent controller snapshot-db /var/lib/ziti-controller/ctrl-backup.db

The controller process writes this file, so the path has to be writable by the ziti-controller user. Give an explicit path. By default the snapshot goes into the raft directory, which you're about to move out of the way.

2. Stop the controller and move the raft directory aside

sudo systemctl stop ziti-controller
sudo mv /var/lib/ziti-controller/raft /var/lib/ziti-controller/raft.old

Keep raft.old until you're happy. It's your rollback.

3. Start the controller and restore it from the snapshot

sudo systemctl start ziti-controller
sudo ziti agent cluster restore-from-db /var/lib/ziti-controller/ctrl-backup.db

4. Check the stored address

sudo ziti agent cluster list

The address should now be tls:controller.example.com:443. You should be able to log in with your existing admin credentials and see all your data.

5. Fix each router's endpoints file

Your routers have :1280 saved in their endpoints file, and when that file exists it takes precedence over ctrl.endpoint in the config. On each router:

sudo systemctl stop ziti-router
sudo rm /var/lib/ziti-router/endpoints.yml
sudo systemctl start ziti-router

With the file gone, the router falls back to ctrl.endpoint in its config.yml, which should be tls:controller.example.com:443. Once it connects, the controller sends it the cluster's address list, and the router writes a new endpoints.yml with :443. It's fine if you see the routers resync their data on that first connection. The rebuilt cluster gets a new cluster ID, so that's expected.

6. Clean up

Once everything is working, remove /var/lib/ziti-controller/raft.old and the snapshot.

If you need to roll back

sudo systemctl stop ziti-controller
sudo rm -rf /var/lib/ziti-controller/raft
sudo mv /var/lib/ziti-controller/raft.old /var/lib/ziti-controller/raft
sudo systemctl start ziti-controller

Let me know how it goes.

Paul

7

Awesome instructions!
That definitely helps me, and I hope it helps others too.
Many thanks!