Change Controller Adresse/Port or migrate to a new controller

Support General Questions
1

So I have a medium sized setup: 1 Controller, 2 Public edge routers and about 12 private routers and 20 clients.

Currently the controller is reachable via ctrl.example.com:1280 and the edge routers via edge1.example.com:3022 and edge2.example.com:3022.

My goal is to change the address or especially the port of the controller to be reachable via port 443 and the edge1 via edge1.example.com:80.

Is there anyway to achieve this without re-enrolling every router and client?
I know that the config.yml of the routers knows the controller address so my plan would be to:
(1) Setup the Controller to listen to 443
(1.1) Setup the edge routers to listen to 80 and advertise this
(2) stop every router, and change the corresponding config.yml the reflect the new controller adress and port
(3) start every router back up.

On the clients... is there even a way to change this? I guess re-enrolling them would be much easier right?

Kind Regards

2

Hi @tekook, welcome to the community and to OpenZiti (and zrok/BrowZer)!

Yes. There is a mechanism in place to "move the controller". You would use the "newAddress" configuration block that's in the web section. It would look like this:

        # newAddress - optional
        # A host:port string which will be sent out as an HTTP header "ziti-new-address" if specified. If the header
        # is present, clients should update location configuration to immediately use the new address for future
        # connections. The value of newAddress must be resolvable both via DNS and validate via certificates
        #newAddress: localhost:1280

After a sufficient amount of time, your clients will all be moved over. If they are all online, I expect that time is not very much. If they are all on PTO/Holiday/Vacation, well then it might be longer... I don't think we've ever documented "how to" move the address yet. If you want, I could come up with the steps, it just might take a bit.

Since you have a relatively small network, in all fairness, it probably IS easy enough to just force them all to reenroll but you're close to the point where it might make more sense to use the newAddress field.

Do you want me to write up the steps? I search Discourse and didn't see it done yet. Lemme know if you'd like that, and I'll come up with the steps...

3

I found the process documented in release notes. Have a look out there and let us know if you need anything else :slight_smile:

4

Oh I missed the "edge" move. For routers, you just need to change the config file and restart them. The 'advertise' address is dynamic. The controller is the challenge as the port is cached in the identity file so you need to 'move' it... Sorry for so many follow-ups :slight_smile:

5

Awesome! Thanks for the instructions and details, I can work with that on my test deployment and move on later to the production deployment! Thanks for the quick reply :slight_smile:

6

@TheLumberjack just for clarification.
Setting newListener and newAddress will open both addresses/ports.
So in my case I deployed via docker, so I need to expose both ports of the controller.

Since I don't intend to change the name in the test-deployment the new certificate generation is not needed right?

7

Correct. If you're not changing the DNS entry, just the port, you don't need to bother with the cert regen. The DNS SANS in the cert will be valid still, so you should be good to go

8

@TheLumberjack one question thou.

web:
  # name - required
  # Provides a name for this listener, used for logging output. Not required to be unique, but is highly suggested.
  - name: all-apis-localhost
    # bindPoints - required
    # One or more bind points are required. A bind point specifies an interface (interface:port string) that defines
    # where on the host machine the webListener will listen and the address (host:port) that should be used to
    # publicly address the webListener(i.e. mydomain.com, localhost, 127.0.0.1). This public address may be used for
    # incoming address resolution as well as used in responses in the API.
    bindPoints:
      #interface - required
      # A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces
      - interface: 127.0.0.1:1280

        # address - required
        # The public address that external incoming requests will be able to resolve. Used in request processing and
        # response content that requires full host:port/path addresses.
        address: 127.0.0.1:1280

        # newAddress - optional
        # A host:port string which will be sent out as an HTTP header "ziti-new-address" if specified. If the header
        # is present, clients should update location configuration to immediately use the new address for future
        # connections. The value of newAddress must be resolvable both via DNS and validate via certificates
        newAddress: localhost:1280

Since I am changing the port, to I need the add another "interface" like this??

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:1280
        address: ctrl.example.com:1280
        newAddress: ctrl.example.com:1281
      - interface: 0.0.0.0:1281
        address: ctrl.example.com:1281
9

Hi @tekook, Yes. You want/need to duplicate the entire "web" entry. Duplicate the entire entry and change the ports. For example, I ran a quickstart and made a second set of apis under a 'new-ports' name:

  - name: new-ports
    bindPoints:
      - interface: 0.0.0.0:2280
        address: sg4:2280
    identity:
      ca:          "/tmp/qs/pki/root-ca/certs/root-ca.cert"
      key:         "/tmp/qs/pki/intermediate-ca/keys/server.key"
      server_cert: "/tmp/qs/pki/intermediate-ca/certs/server.chain.pem"
      cert:        "/tmp/qs/pki/intermediate-ca/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
1 Like
10

Thanks again. Got it working for the clients, after a reconnect the new address/port is saved.

Should the routers upgrade their config.yml automatically? I restarted one of my private routers but the config does not reflect the new change.

The router is run via docker, but that should'nt make any difference right?

  private-er1:
    image: openziti/ziti-router:1.1.2
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role==manager]
    #environment:
      #ZITI_CTRL_ADVERTISED_ADDRESS: ctrl.example.com
      #ZITI_ENROLL_TOKEN: ${ZITI_ENROLL_PRIVATE_ROUTER:-}
      #ZITI_ROUTER_ADVERTISED_ADDRESS: private-er1
      #ZITI_ROUTER_MODE: host
      #ZITI_ROUTER_NAME: private-edge-router

      # *** Bootstrap ***
      #ZITI_BOOTSTRAP: "true"
      #ZITI_BOOTSTRAP_CONFIG: "true"
      #ZITI_BOOTSTRAP_ENROLLMENT: "true"
      #ZITI_AUTO_RENEW_CERTS: "true"
      #ZITI_ROUTER_TYPE: edge
    healthcheck:
      test:
        - CMD
        - ziti
        - agent
        - stats
      interval: 3s
      timeout: 3s
      retries: 50
      start_period: 15s
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - zerotrust-private-net
    volumes:
      - /storage/zerotrust/private-er1:/ziti-router
11

I would not expect the routers to automatically update their config, no. You'll have to either redploy a new router with the new address with it for the controller, or you'll need to update the routers to point to the new port for the control plane.

This docker image I'm not as familar with as @qrkourier. I don't know what it tries to do (or not do) but I would expect the config file within the contain could be modified and the container restarted, but I've not personally tried that. I'm just not entirely sure what sort of lifecyle there might be for it. FWIW, it's coming on a US holiday, so our replies might be delayed for the next few days.

Cheers.

12

Thanks again, and no worries if replies take some time! Enjoy the holiday! :slight_smile:

In the release note is said:

  1. Define the new settings required for the REST APIs (newAddress) and/or control channel (newListener), see below
  2. Restart the controller
  3. Verify existing routers and REST API clients configuration files have updated

That suggested to me the routers would also update by themself, but this is no problem, updating the router files and restarting them was part of my original plan.

I testet everything, the client side works fine and as aspected.
The Routers on the other hand cannot connect to the new port (gives "error connecting ctrl (remote error: tls: internal error)").

I wondered if I need to change the configuration of the controller in the "edge/api" section as well, as the port is defined there as well.
I attached my whole configuration.
controller.yml.txt (11.7 KB)

Does the edge/api section support something like "newAddress" as well?
Or would my next step now be:

  1. verify all clients have updated
  2. change the configuration of the controller to listen to the desired port completely (without newAddress and so on)
  3. Restart the controller
  4. Update all Router Configs
  5. Restart all routers

Kind regards, and don't worry about the a quick reply, this is not a timely matter at all!

13

Yes. The release notes specify a newListener field as well. Are you able to access the routers and change their configurations? It might just be easier on the routers if you do that yourself. What I am not personally sure of, is whether you can declare two control plane listeners. I admit that I've never done that, myself and I didn't test that when I did my own testing before. I'd only tested the newAddress field.

I'll get back around to this on Monday if it hasn't been sorted it out by then. :slight_smile: Have a good weekend!

14

I can access all routers without a problem in both my dev and prod environment.
I followed my steps and changed the controller config to only listen to the new port and changed all routers manually.
Worked like a charm all is up and running on the new port.

15

Sounds like you're up and running then! Great to hear!

16

Okay seems like now I got a different problem with the desktop edge.
The Desktop Edge displays the correct port and url, but the service still connects to the old address.
The iOS Version works fine and connects without a problem.
image
if I take a look at the service log, it still tries to connect to the old port.

[2024-11-30T15:17:53.525Z]   DEBUG ziti-sdk:ziti.c:325 is_api_session_expired() ztx[0] is_api_session_expired[TRUE] - api_session is null
[2024-11-30T15:17:53.525Z]    INFO ziti-sdk:ziti.c:936 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://ctrl.example.com:1280] api_session_status[0] api_session_expired[TRUE]
[2024-11-30T15:17:53.525Z]   DEBUG ziti-sdk:ziti.c:276 ziti_set_auth_started() ztx[0] setting api_session_state[0] to 1
[2024-11-30T15:17:53.525Z]   DEBUG ziti-sdk:ziti.c:360 ziti_stop_api_session_refresh() ztx[0] ziti_stop_api_session_refresh: stopping api session refresh
[2024-11-30T15:17:53.525Z] VERBOSE ziti-sdk:ziti_ctrl.c:143 start_request() ctrl[ctrl.example.com] starting POST[/authenticate?method=cert]
[2024-11-30T15:17:54.575Z]   TRACE tunnel-cbs:ziti_tunnel_ctrl.c:228 process_cmd() processing command[GetMetrics] with data[{"Identifier":"C:\\Windows\\system32\\config\\systemprofile\\AppData\\Roaming/NetFoundry/JuTe_ C-JT01.json"}]
[2024-11-30T15:17:54.575Z]   TRACE ziti-edge-tunnel:ziti-edge-tunnel.c:771 on_write_event() Events message is sent.
[2024-11-30T15:17:54.575Z]   TRACE ziti-edge-tunnel:ziti-edge-tunnel.c:771 on_write_event() Events message is sent.
[2024-11-30T15:17:55.568Z]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[ctrl.example.com] request failed: -4078(connection refused)
[2024-11-30T15:17:55.568Z]    WARN ziti-sdk:ziti.c:1623 api_session_cb() ztx[0] failed to get api session from ctrl[https://ctrl.example.com:1280] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection refused
[2024-11-30T15:17:55.568Z]   DEBUG ziti-sdk:ziti.c:1665 api_session_cb() ztx[0] unhandled error, setting api_session_timer to 5s
[2024-11-30T15:17:55.568Z]   DEBUG ziti-sdk:ziti.c:283 ziti_set_unauthenticated() ztx[0] setting api_session_state[1] to 0
[2024-11-30T15:17:55.568Z]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[ctrl.example.com] clearing api session token for ziti_controller
[2024-11-30T15:17:55.568Z]   DEBUG ziti-sdk:ziti.c:365 ziti_schedule_api_session_refresh() ztx[0] ziti_schedule_api_session_refresh: scheduling api session refresh: 5000ms

I took a look at the profile in C:\Windows\System32\config\systemprofile\AppData\Roaming\NetFoundry, the json still states the old port and address in field "ztAPI".
config.json has the new port but the enrollment json has the old one.

I added back the web section for a client-management on the "old" port, so the client can connect again, so my clients are now back online and they display the old port again.
I tested this section again with the "newAddress" field, the client now displays the new port, but the json file still has the old port.

I checked everything and came to the following conclusion:
The change via "newAddress" works, but only persists while the data-service in running.
After restarting the pc or the service, the gui still displays the new port but tries to connect to the old port. If it connects to the old port and the "newAddress" is not set, the GUI now also reflects the old port.
With the newAddress set, it connects to the new Port but only while the service is running.

So steps to reproduce would be:
(1) configure the controller to have both ports via the web: section.
(2) configure the old port to reflect the "newAddress" field.
(3) restart controller
(4) connect with the desktop-edge and verify the gui states the new port
(5) configure the controller only to have the new port and no newAddress
(6) disable and enable the identity and verify you can still connect via the new port
(7) restart the Ziti Desktop Edge Service (can be done via gui)
(8) verify the gui still has the new port, but conenction is not possible
(8.1) verify the service logs with state the old port.

I think this might be a bug in the ziti-desktop-client.
I can open up an issue on github and provide logs and configuration if you would like to!

And always don't rush yourself with an reply, this it not a production issue or anything important as I can work with an workaround!

17

Update:
controller version: 1.2.2
desktop-edge: App: 2.4.1 Service:1.1.4

18

Ok. So it sounds like the Windows clients are not updating the identity file properly based on the error in the log and your confirmation.

Can you collect the logs from one of the clients that's not wroking as expected by going to the Main Menu -> Feedback button? It will produce a zip file. If you are comfortable sharing that with me via DM here on Discourse or by email to clint at openziti.org, would you share it?

I'll setup a test and try to emulate the situation as well.

19

I can indeed confirm that this doesn't appear to be working on windows any more. I'm sure it's worked in the past, so some bug must have crept into the tunneler on windows. I was also able to confirm the mac tunneler rewrites the file properly.

I'll file a bug for this and we'll get it fixed soon (or figure out exactly why/where the windows tunneler didn't work)

Bug filed here:

20

Sure logs are on the way via mail (couldn't figure out how to send a dm here :smiley:)

Awesome, happy to found a bug :slight_smile: I subscribe to the issue and will get notifications if anything happens. If you need anymore infos or testings I'm happy to help! Reach out to me via github (@tekook) or in this post!

Kind Regards