Questions from a newbie about my planned setup

1

Hello everyone,
I'm still pretty new to OpenZiti, but I have an idea of what my final setup should look like, although I'm not entirely sure from the documentation whether it will work as I envisioned. Finally, I still have a few questions, so sorry for the long thread.

I envision my final setup as follows:

  • In the local network (at home) a private-edge-router runs under router.home.my.domain and listens on port 3022 (default port)
  • Also in the local network, the controller runs under controller.home.my.domain and listens on port 1280 (default port)
  • The ZAC admin interface is installed on the controller and can be accessed via zac.controller.home.my.domain (unfortunately I do not know the default port)
  • Another router is operated as a public-edge router on a VPS at router.my.domain.com and also listens on port 3022
  • New clients (Android, Windows, Linux, ...) can only register in the local network, as the controller and the private router can only be reached from there
  • As soon as the clients have received their configuration, they no longer need the connection to the controller and can also connect to the public edge router from the Internet
  • The ZAC console is therefore only accessible from the local subnet via zac.controller.home.my.domain or via OpenZiti when the connection is active

Does the setup work like this, or are there any suggestions for improvement?

Now to my questions:

  1. Do the router and controller talk via port 1280 and 3022 TCP or HTTP?
  2. Via which port is the ZAC accessible by default?
  3. If the ports speak HTTP, is it possible to operate these ports and the Admin Console behind a reverse proxy so that valid certificates are issued?
  4. Is a graphical OpenZiti client also planned for Linux desktop systems (at best via Flatpak) as with Windows?
  5. I configure practically all my systems with Ansible and Terraform.
    • Is it possible to store config files in a path and the controller “automatically” executes them or removes configurations when files are deleted?
    • Alternatively, is a Terraform module planned for the configuration of the controller?
  6. Can I experiment with a private edge router at home first and add a public edge router later?
  7. Does the public edge router also have to be able to reach the private edge router via port 3022, or is it sufficient for the private edge router to reach the public edge router?

Thanks already for the help, I think the software is super powerful, I just haven't understood it yet :smiley:

\ZzenlD

2

Hi @ZzenlD, welcome to the community and to OpenZiti! (and zrok and Browzer)

This is not correct. The controller has two different APIs, one is specifically for clients -- the client api, one is specifically for managing and maintaining the OpenZiti overlay itself. this is the 'management' API.

At this time (I don't expect this to change fwiw), the controller will always be necessary to be publically available to provide the client api. So "no" wrt to your question:

Does the setup work like this, or are there any suggestions for improvement?

Now -- on to your copious questions!!! :slight_smile:

  1. Do the router and controller talk via port 1280 and 3022 TCP or HTTP?

It depends on how you configure it as to what ports it uses. Those are indeed the default ports at this time. Also, depending on how you deploy the controller it requires one HTTP-based port and one NON-HTTP port. Both of these ports CAN be the same port. Look up "ALPN". Port 1280 can both provide HTTP and non-HTTP... So - that question is complex. Is there a reason you're asking this though? I'd expect it's related to the reverse proxy question below?

  1. Via which port is the ZAC accessible by default?

Well again, "it depends". If you deploy using the zac download/extract mechanism (which is newer-ish), then it's provided on the management API port. By default that's 1280. If you use docker, well then it's entirely up to you as to what that port is... I expect you use the newer-ish deployment and it's on the management API port.

  1. If the ports speak HTTP, is it possible to operate these ports and the Admin Console behind a reverse proxy so that valid certificates are issued?

"valid certificates" is such a loaded term... :slight_smile: You can configure the OpenZiti controller with "alt_server_certs" so that you can deliver a third-party cert from something like LetsEncrypt. If you take a look at the browzer quickstart (it uses the older, bash-based OpenZiti quickstart) you'll see alt_server_certs in use there. You can read some on that here too Conventions | OpenZiti

  1. Is a graphical OpenZiti client also planned for Linux desktop systems (at best via Flatpak) as with Windows?

It's already out there but it's getting longer in the tooth now. Yes it's planned, yes it's out there, yes it "should" work but it's not actively developed lately and might have bugs. It's our future UI though, so yes it's "planned". See releases here Releases · openziti/desktop-edge-ui · GitHub (that's the Github repo obviously, so you can look at the code too if you want)

  1. I configure practically all my systems with Ansible and Terraform.
  • Is it possible to store config files in a path and the controller “automatically” executes them or removes configurations when files are deleted?
  • Alternatively, is a Terraform module planned for the configuration of the controller?

I don't really know what you're trying to do here but OpenZiti is entirely programmable through an API and the ziti CLI performs most (not 100%) of the API operations. So, probably?

  1. Can I experiment with a private edge router at home first and add a public edge router later?

You can - yes, but there's (imo) no big benefit doing this other than being able to operate without the internet. However, be super careful here. When you install the overlay, you get "one chance" to get the PKI right so if you do this, and then try to change the address you're accessing things from it will be "tricky". I know how to do this, because I've worked with OpenZiti for years, but I would really suggest you get your controller and first router setup on the open internet first. It'll just be easier on a new learner like yourself imo. But -- if you want some 'pain', I'm sure you can struggle through making the mistakes... :slight_smile: we do not have a guide for doing this so you're on your own there... :slight_smile:

  1. Does the public edge router also have to be able to reach the private edge router via port 3022, or is it sufficient for the private edge router to reach the public edge router?

A key point of OpenZiti is that "private" resources always reach OUT of your firewall, so you can keep ALL firewalls (even on your home network on individual machines) closed. So yes, it's totally acceptable for a "public" router to have link listeners and "private" routers to be link dialers (not necessarily listeners) for that reason.

Ok -- lots of info to digest there. hth

3

OooOOps i forgot to reply to the reverse-proxy question...

This is also very nuanced. "Sure" you can put it behind a reverse proxy -- like zrok!!! :slight_smile: (you should check out zrok if you haven't already, I bet you'll love what it does too)

BUT -- you can also choose to host the ZAC and ergo the management api on a wholly different IP/port. We call that "splitting the api". Here's a discourse post and couple videos we did two years ago on the topic:

4

Wow! First of all, thank you very much for the quick and detailed answer, I am very impressed :slight_smile:

Now to my understanding:

  • The controller's non-HTTP client API must always be publicly accessible, e.g. at controller.openziti.my.domain
  • I can add ZAC to the controller using the newer download/extract method and make it accessible via the HTTP management API, e.g. mgmt.controller.my.domain/zac
  • The management API is also the API endpoint for ziti-cli
  • Thanks for the hint to install the controller and router publicly first, I was more interested in experimenting locally and getting familiar with it before I set it up publicly accessible

The only things that are still unclear to me are the following:

  1. Wouldn't it be possible to set up the clients (e.g. Windows laptop) locally as the controller is reachable and as soon as the connection is active, the controller client API domain (controller.home.my.domain) is tunneled through OpenZiti to the controller?
  2. Can I put a reverse proxy in front of the management API which terminates the TLS connection and tell the controller this with trusted_proxies? I think that would make the “alt_server_certs” obsolete.
  3. How does a client recognize whether it should use a local router or a publicly accessible router? Does it always check which routers it can reach?

Many thanks for your help already :slight_smile:

5

I would amend this. the Client API, and the "non-HTTP" endpoint must be public. Often that's the same port, but the older bash-based quickstarts put them on separate ports.

Yes - certainly. As soon as you left that network, "everything OpenZiti" would stop working as you could no longer contact the controller. The clients ask the controller to establish a connection, they must be able to reach the controller.

No. The controller must terminate the TLS connection. I doubt that will ever change but who knows, maybe... that's how it is now.

Clients will attempt to connect to (by default) 3 routers. Currently, it will use whatever router returns the response fastest as the primary router. Local traffic is almost guaranteed to be fastest, ergo it'll use the local router. It'll still connect to other routers for failover reasons though.

6

If I understand this correctly, the controller does not need several open ports as before, but only port 1280 for any communication (client API, management API, ...). Isn't this potentially a security risk if the management API is publicly accessible?

Further down you wrote that the clients always keep active connections to all reachable routers as failover.
My understanding now is that if the client leaves the local network, it uses the active connection to the publicly accessible router and the controller can still be reached via this router, since the controller's domain is tunneled to the controller via the public router.
Do you understand what I mean or am I on the wrong track?

7

It's a risk, yes. For starters, I would urge you to start out in this deployment model (as I also did just the other day on a different forum post here All-in-one docker compose - #11 by TheLumberjack)

Once you are comfortable with OpenZiti and understand things and have it working, you could/should "split" the management API away from the internet entirely. There are several posts on the forum about this, doc, and videos to reference. :slight_smile:

up to three by default - yes. when you leave one network or a router becomes unresponsive/unreachable, the client will failover to the next router with the lowest response time.

8

Okay, but to execute the failover, the client needs a connection to the controller, right?

The client cannot execute the failover without a controller connection and then connect to the controller via the new router?

For a better understanding:

  • My thought would be to include the controller domain as ziti-service as well and thus make it reachable via the ziti network
  • If a router fails, the client uses the already established connection to the public router
  • The client now tries to reach the controller at controller.home.my.domain from the Internet, whereupon OpenZiti forwards the request to the controller via the Ziti network
9

To make any connection, the client needs to connect to the controller. Clients never connect to the controller through routers. they always go directly to the client API.

My thought would be to include the controller domain as ziti-service

This is an inception-type issue. You can't do this because to dial that serice, you would have had to authenticate to the controller and initiated a connection request allowing you to acces the service.

10

After a lot of back and forth, and a lot of confusion about the many different ports and start options of OpenZiti, I have now tried it with the All-In-One Docker-Compose.

I start OpenZiti in a rootless podman with the following docker-compose file:

version: "3.9"
services:
  openziti:
    image: docker.io/openziti/ziti-controller:1.1.15
    hostname: openziti 
    security_opt:
      - no-new-privileges
    ports:
      - 1280:1280 #Controller-Traffic needs public access
      - 3022:3022 #Router-Traffic needs public access
      - 8080:8080 #Management-API + ZAC
    user: 1000
    volumes:
      - ./openziti/data:/home/ziggy:z
    labels:
      - io.containers.autoupdate=registry
    environment:
      PFXLOG_NO_JSON: true
      ZITI_CTRL_ADVERTISED_ADDRESS: openziti.my.domain 
      ZITI_CTRL_EDGE_ADVERTISED_ADDRESS: openziti.my.domain
      ZITI_ROUTER_ADVERTISED_ADDRESS: openziti.my.domain
      ZITI_ROUTER_NAME: openziti.my.domain
      ZITI_PWD: secure_password
    entrypoint:
      - bash
      - -euc
      - |
        ZITI_CMD+=" --ctrl-address $${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}"\
        " --ctrl-port $${ZITI_CTRL_ADVERTISED_PORT:-1280}"\
        " --router-address $${ZITI_ROUTER_ADVERTISED_ADDRESS:-$${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}}"\
        " --router-port $${ZITI_ROUTER_PORT:-3022}"\
        " --password $${ZITI_PWD:-admin}"
        echo "DEBUG: run command is: ziti $${@} $${ZITI_CMD}"
        exec ziti "$${@}" $${ZITI_CMD}
    command: -- edge quickstart --home /home/ziggy/quickstart
    networks:
      - proxy
    restart: always

networks:
  proxy:
    external:
      name: proxy

However, for some reason the router cannot connect to the controller at openziti.my.domain:1280. The logs are:

[openziti] | [   3.950]    INFO ziti/controller/server.NewController: edge controller instance id: cm2bhiuzk000101pqjagqf7az
[openziti] | [   3.950]    INFO ziti/controller.(*Controller).RegisterXmgmt: adding xmgmt *server.submgmt, enabled? true
[openziti] | [   3.950]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[openziti] | [   3.969]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {frequency=[5s] sessionTimeout=[30m0s]} session enforcer configured
[openziti] | [   3.970]    INFO ziti/controller/server.(*Controller).Run: starting edge
[openziti] | [   3.970]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[1] maxWorkers=[16] idleTime=[10s] poolType=[pool.listener.ctrl] minWorkers=[1]} starting goroutine pool
[openziti] | [   3.970]    INFO ziti/controller/server.(*Controller).checkEdgeInitialized: edge initialized
[openziti] | [   3.970]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[openziti] | [   4.074]    INFO ziti/controller/zac.ZitiAdminConsoleFactory.New: initializing ZAC SPA Handler from /ziti-console
[openziti] | [   4.074]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:1280 for server client-management with APIs: [edge-management edge-client fabric zac]
[openziti] | [   4.118]    INFO ziti/controller/network.(*Network).Run: started
[openziti] | timed out waiting for controller: https://openziti.my.domain:1280
[openziti] | Environment left intact at: /home/ziggy/quickstart

If I leave it at the default values under Controller is accessible under “quickstart” it works. Problem: My clients cannot reach the controller via “quickstart”, but need an FQDN or IP.

Does anyone here have a solution or an idea where the problem could lie?

Thanks :slight_smile:

11

What if you provide a network alias matching the controller's FQDN? You may have omitted this feature of the example compose file when you transposed it for Podman. Setting an alias allows clients inside the same Docker network to shortcut to their destination, avoiding hairpinning.

I'm excited to hear if it works well in Podman! BTW, there is no need for rootless mode (running as uid 0 mapped to non-root on the Podman host) because you're already running as non-root uid 1000.

services:
  openziti:
    image: docker.io/openziti/ziti-controller:1.1.15
    hostname: openziti 
    security_opt:
      - no-new-privileges
    ports:
      - 1280:1280 #Controller-Traffic needs public access
      - 3022:3022 #Router-Traffic needs public access
      - 8080:8080 #Management-API + ZAC
    user: 1000
    volumes:
      - ./openziti/data:/home/ziggy:z
    labels:
      - io.containers.autoupdate=registry
    environment:
      PFXLOG_NO_JSON: true
      ZITI_CTRL_ADVERTISED_ADDRESS: openziti.my.domain 
      ZITI_CTRL_EDGE_ADVERTISED_ADDRESS: openziti.my.domain
      ZITI_ROUTER_ADVERTISED_ADDRESS: openziti.my.domain
      ZITI_ROUTER_NAME: openziti.my.domain
      ZITI_PWD: secure_password
    entrypoint:
      - bash
      - -euc
      - |
        ZITI_CMD+=" --ctrl-address $${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}"\
        " --ctrl-port $${ZITI_CTRL_ADVERTISED_PORT:-1280}"\
        " --router-address $${ZITI_ROUTER_ADVERTISED_ADDRESS:-$${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}}"\
        " --router-port $${ZITI_ROUTER_PORT:-3022}"\
        " --password $${ZITI_PWD:-admin}"
        echo "DEBUG: run command is: ziti $${@} $${ZITI_CMD}"
        exec ziti "$${@}" $${ZITI_CMD}
    command: -- edge quickstart --home /home/ziggy/quickstart
    networks:
      proxy:
        aliases:
          - ${ZITI_CTRL_ADVERTISED_ADDRESS}
    restart: always

networks:
  proxy:
    external:
      name: proxy
12

I have found the error, the FQDN of the controller must be added as exta_hosts, then it works. The docker-compose now looks like this:

---
version: "3.9"
services:
  openziti:
    image: docker.io/openziti/ziti-controller:1.1.15
    hostname: openziti 
#    read_only: true
    security_opt:
      - no-new-privileges
    ports:
      - 1280:1280 #Controller-Traffic needs public access
      - 3022:3022 #Router-Traffic needs public access
      - 8080:8080 #Management-API + ZAC
    user: 1000
    volumes:
      - ./openziti/data:/home/ziggy:z
    labels:
      - io.containers.autoupdate=registry
    environment:
      PFXLOG_NO_JSON: true
      ZITI_CTRL_ADVERTISED_ADDRESS: controller.openziti.my.domain
      ZITI_ROUTER_ADVERTISED_ADDRESS: router.openziti.my.domain
      ZITI_ROUTER_NAME: router.openziti.my.domain
      ZITI_PWD: secret_password 
    entrypoint:
      - bash
      - -euc
      - |
        ZITI_CMD+=" --ctrl-address $${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}"\
        " --ctrl-port $${ZITI_CTRL_ADVERTISED_PORT:-1280}"\
        " --router-address $${ZITI_ROUTER_ADVERTISED_ADDRESS:-$${ZITI_CTRL_ADVERTISED_ADDRESS:-quickstart}}"\
        " --router-port $${ZITI_ROUTER_PORT:-3022}"\
        " --password $${ZITI_PWD:-admin}"
        echo "DEBUG: run command is: ziti $${@} $${ZITI_CMD}"
        exec ziti "$${@}" $${ZITI_CMD}
    command: -- edge quickstart --home /home/ziggy/quickstart
    extra_hosts:
      - "controller.openziti.my.domain:127.0.0.1"
    networks:
      - proxy
    restart: always

networks:
  proxy:
    external:
      name: proxy

It also worked:

  • Provide management API and ZAC on a separate port
  • Forward all three ports via Traefik via a TCP proxy, so port 443 can always be used as follows:
    • Controller (Public access): controller.openziti.my.domain:443
    • Controller (Management, ZAC): mgmt.openziti.my.domain:443
    • Router (Public access): router.openziti.my.domain:443

However, I still have three final problems:

  1. The accesses via ZAC are very sluggish and the following error messages can be found a lot in the log:
[openziti] | [  29.505]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:46914] error=[EOF]} handshake failed
[openziti] | [  29.512]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:46922]} handshake failed
[openziti] | [  29.513]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:46916] error=[EOF]} handshake failed
[openziti] | [  32.551]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:46930] error=[EOF]} handshake failed
[openziti] | [  32.552]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:46932]} handshake failed
[openziti] | [  56.118]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [h2 http/1.1]] remote=[10.89.1.2:56322]} handshake failed
[openziti] | [  56.119]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.2:56332] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [  62.278]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.2:44396] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [  70.518]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.2:49422] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 103.428]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57690] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.435]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57702] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.438]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57716] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.442]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57730] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.446]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57736] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.451]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: first record does not look like a TLS handshake] remote=[10.89.1.21:57738]} handshake failed
[openziti] | [ 103.456]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57752] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.460]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57766] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.465]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:57770] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 103.469]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: first record does not look like a TLS handshake] remote=[10.89.1.21:57774]} handshake failed
[openziti] | [ 108.624]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:49842] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 112.475]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:49852] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 113.181]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:49854] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 127.819]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:59142] error=[EOF]} handshake failed
[openziti] | [ 154.173]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[10.89.1.2:57652] error=[remote error: tls: unknown certificate authority]} handshake failed
[openziti] | [ 218.600]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33378] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.605]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33384] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.609]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33386] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.614]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33402] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.618]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33412] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.623]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33426] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.627]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33442] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.631]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33448] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.636]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33456] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 218.640]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33462] error=[tls: first record does not look like a TLS handshake]} handshake failed
[openziti] | [ 225.931]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33468] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 235.734]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [h2 http/1.1]] remote=[10.89.1.21:37284]} handshake failed
[openziti] | [ 236.702]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.21:33962] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 256.278]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.2:46270] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 267.451]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [h2 http/1.1]] remote=[10.89.1.2:43930]} handshake failed
[openziti] | [ 275.371]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {remote=[10.89.1.2:43940] error=[not handler for requested protocols [h2 http/1.1]]} handshake failed
[openziti] | [ 316.465]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[10.89.1.2:44502] error=[EOF]} handshake failed
[openziti] | [ 344.652]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:42140] error=[EOF]} handshake failed
[openziti] | [ 344.663]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:42146]} handshake failed
[openziti] | [ 344.664]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:42148]} handshake failed
[openziti] | [ 352.823]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[10.89.1.21:46812] error=[remote error: tls: unknown certificate authority]} handshake failed
[openziti] | [ 363.947]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[remote error: tls: unknown certificate authority] remote=[10.89.1.21:53048]} handshake failed
[openziti] | [ 401.836]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:41940] error=[EOF]} handshake failed
[openziti] | [ 401.837]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:41998] error=[EOF]} handshake failed
[openziti] | [ 401.848]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:41972]} handshake failed
[openziti] | [ 401.848]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:41956] error=[EOF]} handshake failed
[openziti] | [ 401.855]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:41986]} handshake failed
[openziti] | [ 410.945]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:51974] error=[EOF]} handshake failed
[openziti] | [ 586.952]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:54442] error=[EOF]} handshake failed
[openziti] | [ 586.952]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:54456] error=[EOF]} handshake failed
[openziti] | [ 586.959]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:54426]} handshake failed
[openziti] | [ 586.959]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:54412] error=[EOF]} handshake failed
[openziti] | [ 586.960]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:54468] error=[EOF]} handshake failed
[openziti] | [ 586.966]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:54478] error=[EOF]} handshake failed
[openziti] | [1010.759]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {remote=[10.89.1.2:52082] error=[remote error: tls: unknown certificate authority]} handshake failed
[openziti] | [1032.682]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: unknown certificate authority] remote=[10.89.1.2:49438]} handshake failed
[openziti] | [1391.993]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8080]: {error=[EOF] remote=[10.89.1.2:58554]} handshake failed

It looks as if ZAC cannot properly log on to the management API (on port 8080 in the container) “error=[EOF]} handshake failed”.

  1. Currently port 1280 and 3022 are still advertised, but this always fails. What adjustment is necessary so that external clients call the respective domains via 443 and not via 1280 or 3022?

  2. The OpenZiti root CA automatically sets organization, etc. to Charlotte, NetFoundry, ... this is not very nice. Would it be possible to parameterize this via environment variables in a future version?

As soon as I have everything running, I will share my working docker-compose.yml, ctrl.yml, and traefik.yml here, in the hope that it will help others :slight_smile:

13

Yes, adding an extra_hosts DNS record to your external Docker bridge should have the same effect as adding the alias to the network spec on the quickstart container named "openziti."

As for re-issuing the controller's server leaf with customized DN, you'll need these usage hints from the CLI:

❯ ziti pki create server --help
Creates new Server certificate (signed by previously created Intermediate-chain)

Usage:
  ziti pki create server [flags]

Aliases:
  server, s

Flags:
      --allow-overwrite        Allow overwrite existing certs
      --ca-name string         Name of Intermediate CA (within PKI_ROOT) to use to sign the new Server certificate (default "intermediate")
      --curve string           If set an EC private key is generated and -private-key-size is ignored, options: P224, P256, P384, P521
      --dns strings            DNS name(s) to add to Subject Alternate Name (SAN) for new Server certificate
      --expire-limit int       Expiration limit in days (default 365)
  -h, --help                   help for server
      --ip strings             IP addr(s) to add to Subject Alternate Name (SAN) for new Server certificate
      --key-file string        Name of file (under chosen CA) containing private key to use when generating Server certificate
      --max-path-len int       Intermediate maximum path length (default -1)
      --pki-root string        Directory in which PKI resides
      --private-key-size int   Size of the RSA private key, ignored if -curve is set (default 4096)
      --server-file string     Name of file (under chosen CA) in which to store new Server certificate and private key (default "server")
      --server-name string     Common Name (CN) to use for new Server certificate (default "NetFoundry Inc. Server")
      --spiffe-id string       Optionally provide the path portion of a SPIFFE id. The trust domain will be taken from the signing certificate.

Global Flags:
      --pki-country string               Country (default "US")
      --pki-locality string              Locality/Location (default "Charlotte")
      --pki-organization string          Organization (default "NetFoundry")
      --pki-organizational-unit string   Organization unit (default "ADV-DEV")
      --pki-province string              Province/State (default "NC")

e.g.,

compose exec openziti ziti pki create server \
--pki-root "/home/ziggy/quickstart/pki" \
--server-name "my server" \
--ca-name "intermediate-ca" \
--pki-country "dnC" \
--pki-province "dnS" \
--pki-locality "dnL" \
--pki-organization "dnO" \
--pki-organizational-unit "dnOU" \
--dns "localhost,controller.openziti.my.domain" \
--ip "127.0.0.1,::1" \
--allow-overwrite

where

openssl s_client -connect 127.0.0.1:1280 <>/dev/null \
|& openssl x509 -noout -subject

gets

subject=C = dnC, L = dnL, O = dnO, OU = dnOU, CN = my server

EDIT: updated to add -server-name "my server" to set CN

14

This command can be used to customize the server certificate, but the root-ca and intermediate-ca still contain the default values from NetFoundry.
Is it also possible to specify Country, Province, Organization, ... directly when creating these two CAs?

In case other people are also reading this thread, here are two more hints:

  1. The error message “handshake failed”, which occurs en masse, is due to the fact that ZAC logs on to the Management API with a username/password and not with a certificate. However, I have not yet found a way to get rid of the error message or to suppress it.
  2. The access of all APIs via own domains and port 443 behind a reverse proxy works perfectly in the meantime and external clients can log in successfully. I will share my configuration files here later.

@qrkourier I have one more question:
Can the ctrl.yaml be customized so that browser calls to the management API are automatically redirected to /zac?

15

Yes. Run ziti pki --help the same way as the exec example for more usage hints.

I raised this GitHub issue to track fixing this problem that keeps causing confusion: spurious handshake failed errors are actually expected, not errors · Issue #2486 · openziti/ziti · GitHub

Proxying to the controller's REST APIs works for specific things but not most. You can undoubtedly use TCP LB w/ passthrough TLS, but terminating TLS on an LB breaks mTLS (i.e., client cert auth), which is central to how Ziti works.

No, the ZAC request for the login page must have URL path /zac/. Requests for the management API always have base URL path /edge/management/v1. You could proxy the management API and manipulate URL paths in upstream requests if you are not using cert auth for console or CLI.

16

First of all, thank you very much for the good answers, they really help me.

Yes, that may have been explained somewhat incomprehensibly. I have configured a TCP proxy with tls passtrough. This means that the controller's certificates are still issued and the connections are only terminated there. mTLS should therefore not be interrupted. Especially since, as I understand it, no Ziti client would connect to a router or controller if its certificate could not be reached.

1 Like
17

I have now tested OpenZiti extensively for a few days and can say that it runs very stably even under rootless podman.

But I still have a few things (sorry):

  1. There is a reverse proxy in front of the applications that automatically forwards from HTTP to HTTPS. How can I tell OpenZiti to make an interception for ports 80 and 443 of a domain and forward both ports to the respective ports of the service? In this case, this is required for the automatic HTTPS redirect. I don't want to have to create multiple services for this.

  2. Can I check in OpenZiti as a posture check whether the enddevice has activated the device lock (password/pin/fingerprint....)?

And finally, two suggestions for improvement:

  1. Unfortunately I have not yet managed to provide the Root-CA and Intermediate-CA with their own values. It would be nice if these values could be transferred via ENV-variables in future and the controller could automatically apply them when setting up the PKI. This reduces the manual effort.

  2. I think if alpine-linux was used as the basis for the containers, the overall image would probably be much smaller.

Many thanks for the great software :slight_smile:

18

You have Ziti host config(s) that target a reverse proxy, correct? This means that when the traffic exits the Ziti network it flows to a reverse proxy.

You can have multiple ports in a Ziti config. You'll need to define the ports on both sides of the service: intercept config and host config.

intercept.v1 example

{
    "portRanges": [
      {
        "high": 80,
        "low": 80
      },
      {
        "high": 443,
        "low": 443
      }
    ],
    "addresses": [
      "website.ziti.example.com"
    ],
    "protocols": [
      "tcp"
    ]
}

matching host.v1 example

{
    "protocol": "tcp",
    "address": "reverseproxy.ziti.example.com",
    "forwardPort": true,
    "allowedPortRanges": [
      {
        "high": 80,
        "low": 80
      },
      {
        "high": 443,
        "low": 443
      }
    ],
    "httpChecks": [],
    "portChecks": []
}

I didn't find any posture checks that would accomplish that specifically.

We talked earlier in this thread about one approach to specifying the distinguished name (DN) that appears in the x509 subject property. That didn't work, or you wanted something different?

I see what you mean about using env vars for this. I have an idea how to do it, but it will only work for Linux and Docker deployments. Are you still using Docker to self-host your Ziti controller?

That might work. I used the RedHat universal base image because it's tiny, uses glibc, has comparatively few CVEs, is a versatile, distro-full/general-purpose image, and is required for RedHat certification.

Switching to a MUSL-based Alpine distro, or distroless, or a zero-cve/hardened image like ChainGuard are all interesting for related reasons.

At this moment, the general purpose UBI foundational image is solving specific problems and avoiding a proliferation of bespoke images, e.g., distroless.

Here's the size progression from the UBI, then ziti-cli, then ziti-controller, each is built FROM the previous source image. There's may an opportunity to minimize ziti-cli, because the executable delivered by that image is only ~126MB, yet the image is ~411MB larger than the UBI for some reason, meaning there's ~285MB of possible bloat.

❯ for IM in registry.access.redhat.com/ubi9/ubi-minimal:latest openziti/ziti-{cli,controller}:1.1.15 ; do docker image ls --all $IM ; done | column -t
REPOSITORY                                   TAG     IMAGE         ID  CREATED  SIZE  
registry.access.redhat.com/ubi9/ubi-minimal  latest  dc90caed48cf  5   weeks    ago   99.2MB
REPOSITORY                                   TAG     IMAGE         ID  CREATED  SIZE  
openziti/ziti-cli                            1.1.15  650c736a64b1  3   weeks    ago   510MB
REPOSITORY                                   TAG     IMAGE         ID  CREATED  SIZE  
openziti/ziti-controller                     1.1.15  e897996c0199  3   weeks    ago   647MB

❯ docker image history openziti/ziti-cli:1.1.15 | grep -v 0B
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
<missing>      3 weeks ago   COPY ./dist/docker-images/ziti-cli/bashrc /h…   829B      buildkit.dockerfile.v0
<missing>      3 weeks ago   RUN |7 ARTIFACTS_DIR=./release DOCKER_BUILD_…   11.7kB    buildkit.dockerfile.v0
<missing>      3 weeks ago   RUN |7 ARTIFACTS_DIR=./release DOCKER_BUILD_…   132MB     buildkit.dockerfile.v0
<missing>      3 weeks ago   COPY ./release/amd64/linux/ziti /usr/local/b…   132MB     buildkit.dockerfile.v0
<missing>      3 weeks ago   RUN |7 ARTIFACTS_DIR=./release DOCKER_BUILD_…   2.77kB    buildkit.dockerfile.v0
<missing>      3 weeks ago   COPY ./LICENSE /licenses/apache.txt # buildk…   11.3kB    buildkit.dockerfile.v0
<missing>      3 weeks ago   COPY /opt/bitnami/kubectl/bin/kubectl /usr/l…   56.4MB    buildkit.dockerfile.v0
<missing>      3 weeks ago   RUN |7 ARTIFACTS_DIR=./release DOCKER_BUILD_…   90.5MB    buildkit.dockerfile.v0
<missing>      5 weeks ago   /bin/sh -c mv -fZ /tmp/ubi.repo /etc/yum.rep…   99.2MB

Besides the ziti binary's 132MB layer, the biggest contributors are kubectl (56.4MB), Linux packages python3.11 python3.11-pip tar bash-completion vim-minimal less shadow-utils jq findutils hostname (90.5MB). The stuff we're adding in this ziti-cli image add up to ~146MB, so it's not immediately clear where the other ~139MB came from. It's odd that the image history assigns the same size to the RUN /bin/sh -c chmod 0755 /usr/local/bin/ziti layer as the COPY layer that installs the ziti executable. Changing the filemode shouldn't bloat the image, but it's almost the same size as the missing difference, so I wonder if that's it. :thinking:

19

Yes, thank you very much, that was the missing part. I had created the first services manually via ZAC and there I could only enter one port in the host config :slight_smile:

Neither do I, but perhaps that would be a suggestion for another Posture Check.
I think it is very desirable to ensure that an enddevice has active access protection enabled. Otherwise, anyone could unlock the device and access services in the Ziti network (which is not quite Zero Trust :D)

Yes, I use containers powered by podman on all systems and they run perfectly so far. Running the CA with my own data is not priority 1 now but would be “nice to have” in the future.

That was just a suggestion for improvement from me. For me personally, the current image also fits, although of course the smaller the better :slight_smile:

1 Like
20

Not exactly what you are asking for, but see Posture Checks | OpenZiti, ("MFA Posture Checks also support forcing a client to re-submit a valid TOTP on timeout, after locking/unlocking a device, or waking a device from sleep.")