No Controllers Available

[Damien]

Hi,

After changing web browser (or cleaning the browser local storage), the ZAC do not works anymore:

So I declared the environment variable first in service.env which doesn't worked, in bootstrap.env and then hardcoded in entrypoint.bash because this is the script executed at the controller startup:

The problem persists, the URL specified in the variable matches the bindPoint's address of the ziti controller.

The ziti-controller.service was restarted each time I edited files, and browser local storage deleted before reloading the zac page.

How can I fix it, please?

[TheLumberjack]

see Community Update: Adding controller URL's for NodeJS hosted ZAC instances

[Damien]

I already saw this post and not resolved my issue
openziti-console is already the newest version (3.9.1).
my zac is 3.9+ and openziti hosted anywhere

I tried ziti edge login, not working even after restarting ziti-controller.service

[TheLumberjack]

Are you saying you access your ZAC at $controller.url/zac and you still get this popup??? Can you try a private window? Would you be willing to share the url with me either here or DM?

I haven't experienced this myself. Is there anything about your setup that you can share? Have you split the management API maybe?

@rgalletto fyi - is there any way this could happen for controller-hosted ZAC?

[Damien]

Yes exactly, zac running on a separate port of the API but on the same host.
Private window also not working, sharing my URL by DM

api management not split:

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:48440
        address: hidden
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }

  - name: zac-console
    bindPoints:
      - interface: 0.0.0.0:48442
        address: 0.0.0.0:48442
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: zac
        options:
          location: /opt/openziti/share/console
          indexFile: index.html

[TheLumberjack]

@Damien

Yes exactly, zac running on a separate port of the API but on the same host.

Ah. That makes sense as that will be what triggers this behavior. You're saying this configuration worked before? That confuses me. I would have expected you to receive an error.

I would say you should downgrade your ZAC to a pre 3.9 for now. I think we'll have to test this situation.

[TheLumberjack]

Oh you could also just put the ZAC on the same port as the management API. That will also fix this for you and let you stay on 3.9.x

[Damien]

This configuration worked properly when the local storage of my browser with all parameters was set. Once flushed, do not worked anymore.

Downgraded to 3.9.0 and same problem, flushed local storage, tried in private window...

root@openziti-controller:~# apt install openziti-console=3.9.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be DOWNGRADED:
  openziti-console
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 10.5 MB of archives.
After this operation, 1024 B disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 https://packages.openziti.org/zitipax-openziti-deb-stable debian/main amd64 openziti-console amd64 3.9.0 [10.5 MB]
Fetched 10.5 MB in 2s (5193 kB/s)           
dpkg: warning: downgrading openziti-console from 3.9.1 to 3.9.0
(Reading database ... 20293 files and directories currently installed.)
Preparing to unpack .../openziti-console_3.9.0_amd64.deb ...
Unpacking openziti-console (3.9.0) over (3.9.1) ...
Setting up openziti-console (3.9.0) ...
root@openziti-controller:~# systemctl restart ziti-controller.service 
root@openziti-controller:~#

Also rebooted my controller where the ZAC is hosted in order to try to delete some residual cache files, but same problem again..

[Damien]

Yes it was in this configuration in the past, but I do not want to make my ZAC internet facing for security reasons...

[TheLumberjack]

Ok, the other option is to also provide the managment API on the same "internal" port. The ZAC probes the same host:port, so you can either put zac on the public port, or host the management api on just the private port which sounds like what you were trying to do anyway.

Move

      - binding: edge-management
        options: { }

to the zac-console and you should be good to go

[Damien]

I want to make accessible the ZAC only through a ziti service or the LAN, how I configured it:

- name: zac-console
    bindPoints:
      - interface: 0.0.0.0:48442
        address: 192.168.X.X:48442
        address: console.ziti:443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: zac
        options:
          location: /opt/openziti/share/console
          indexFile: index.html
      - binding: edge-management

Now it is working. The ZAC is only accessible via the LAN address or the console.ziti address. I tried to access it via my public IP and was instantly reset because I do not match addresses in the config file.

I also before connecting to my controller, delete the local storage, refresh the page and go to private window and I no longer receive the error message.

Maybe there is an issue in the code where the ZAC has difficulties to connect to the controller where not on the same port? Or maybe it is designed to, but I don't think so because I have now two controller API endpoint: one on the normal port and one on the zac port.

[TheLumberjack]

@Damien

Or maybe it is designed to,

Yes it was designed to work this way. It actually surprised us to learn that the zac on a different port DID work. that was unexpected.

Glad you got things working and your setup is now safer as the management API is not available/attackable via the open internet.

[Damien]

Yes! That's only my ZAC which is not available on the open internet, the management API is exposed but I do not have choice if I want to be connected anywhere.

Thank you a lot for your fast support by the way!

[TheLumberjack]

You can remove the fabric and mgmt api from the public ip. The only api's you need on the public ip is the client api and the oidc api (which you'll use in the future)

[Damien]

Oh okay! And what is, in the bindPoints, the address a client would use to contact the mgmt and fabric API, for hardening?

[TheLumberjack]

using your config from before i would change this section:

@Damien

      - interface: 0.0.0.0:48442
        address: 192.168.X.X:48442

I'd bind the interface to 192.168.X.X (not all interfaces) and then any time you wanted to manage/maintain the controller, you would need to be on the local network that would allow you to access 192.168.X.X.

You could ALSO choose to limit it to 120.0.0.1 and force anyone wanting to manage/maintain the controller to be on the same machine.

Recently I covered using a router colocated with the controller to allow for exactly this, and thus you can only access/manage/maintain the controller if you're ON the openziti overlay itself here Making ZAC and management API accessible only through service

You have to bootstrap it with ziti itself, but once you do you could then configure OpenZiti as shown above and in that post and exclusively manage ziti -- using ziti...

[Damien]

Okay now I see how it works.

Since I have my controller behind a NAT, I have only one interface to listen to. So I changed the mgmt/fabric port on a port which is not exposed to the WAN, and accessible only with specific addresses:

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:48440 # Internet facing
        address: hidden
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }

  - name: zac-console
    bindPoints:
      - interface: 0.0.0.0:48442  # Only LAN
        address: 192.168.X.X:48442
        address: console.ziti:443
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms 
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: zac
        options:
          location: /opt/openziti/share/console
          indexFile: index.html
      - binding: edge-management

  - name: mgmt-fabric
    bindPoints:
      - interface: 0.0.0.0:48443  # Only LAN
        address: 127.0.0.1:48443        # Local management
        address: 192.168.X.X:48443    # Admin Network
    identity:
      ca:          "pki/root/certs/root.cert"
      key:         "pki/intermediate/keys/server.key"
      server_cert: "pki/intermediate/certs/server.chain.pem"
      cert:        "pki/intermediate/certs/client.chain.pem"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
      - binding: fabric

Thanks to your advice I managed to hide the edge-management and fabric endpoints to the LAN only.

Recently I covered using a router colocated with the controller to allow for exactly this, and thus you can only access/manage/maintain the controller if you're ON the openziti overlay itself here Making ZAC and management API accessible only through service

Yes I read this post before sending mine, that is how I minded to manage ziti too, for now I do not use command line but can be useful for automation. Currently, only managing through the ZAC through ziti (console.ziti) network and as backup if for X reasons ziti network do not work, through the LAN.