Setting up a simple controller + edge router + zac on docker

Hi,

I've a arm64 Oracle Cloud VM running docker and I'm trying to get a simple Controller + Edge router + ZAC runnning on docker containers.

I've managed to get the controller running:

sudo docker run --name openziti_controller --user 0 --rm --privileged --network openziti --platform linux/arm64 \
  -v /opt/openziti_controller:/ziti-controller \
  -e ZITI_AUTO_RENEW_CERTS=true \
  -e ZITI_BOOTSTRAP=true \
  -e ZITI_BOOTSTRAP_CONFIG=true \
  -e ZITI_BOOTSTRAP_DATABASE=true \
  -e ZITI_BOOTSTRAP_PKI=true \
  -e ZITI_CTRL_ADVERTISED_ADDRESS=<PUBLIC_IP_ADDRESS> \
  -e ZITI_CTRL_ADVERTISED_PORT=1280 \
  -e ZITI_PWD=super_secret \
  -p 1280:1280 \
  openziti/ziti-controller:1.3.3 run config.yml

The controller is running, but when I try to get a JWT to enroll an edge router I have this error:

[root@919837b1d16e ~]# ziti edge create edge-router "router1" \
   --jwt-output-file=./router1.jwt
error: error creating edge-routers instance in Ziti Edge Controller at https://localhost:1280/edge/management/v1. Status code: 500 Internal Server Error, Server returned: {
    "error": {
        "cause": {
            "code": "UNHANDLED",
            "message": "could not get enrollment signer: could not determine enrollment signer: could not find a configured server certificate that matches hostname [79.72.31.69] in root controller identity nor in xweb identities"
        },
        "code": "UNHANDLED",
        "message": "An unhandled error occurred",
        "requestId": "Jfptdo13F"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}

Welcome back, @brandi!

Here's a video tour of running the controller (comes with the console) and a router in Docker: https://youtu.be/9kkcLYdWwPE?si=RaISQoLJotAHqRgU

00:35 Security Groups (firewall rules) 01:50 Install updates 02:20 Install Docker 03:20 Docker group 04:30 Docker Compose 06:00 Configure Controller 10:00 Run Controller 10:48 Install ziti CLI and login 12:45 Configure Router 15:20 Run Router

You aren't required to use Compose. I like it because I can save my settings in a file and keep track of changes with Git.

In your Oracle cloud firewall settings, ensure you have two TCP ports available for Ziti and that you set those two ports as the ZITI_CTRL_ADVERTISED_PORT and ZITI_ROUTER_PORT variables.

BTW, it's not necessary to run the controlelr as root if you'd prefer not to for security. The router sometimes needs to run as root, but only if you're using it as a transparent proxy. That's so it can manage iptables rules and DNS.

Yup, I made that command from the docker compose on the documentation. I'm actually deploying my containers using Terraform. I kinda prefer it over compose.

I actually have the ZITI_CTRL_ADVERTISED_PORT opened :


ZITI_ROUTER_PORT I did not add it yet, because the router isn't deployer yet. Do I have to when deploying the controller container ?

You can wait to add the router port to your firewall. It's not needed until a Ziti identity tries to dial a service.

Do I need to run

ziti edge create edge-router "router1" \
   --jwt-output-file=./router1.jwt

from outside of the controller's container ?

I'm not sure why it is not working :confused: I've also tried to create the edge router from ZAC, but encountered the same error on the UI

You can run your ziti administrative commands on the Docker host or inside any container that can reach the controller's management API, which served on the main TCP port by default.

I like to install the openziti package on the Docker host so I don't have to exec it inside a container. It's more convenient.

Here's the part of the video about installing the CLI on the docker host: https://www.youtube.com/watch?v=9kkcLYdWwPE&t=648s

Yup I got to that part, but idk for whatever reason, it's not working propely on my end lol

{"file":"github.com/openziti/ziti/controller/env/appenv.go:333","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getEnrollmentTlsCert","hostnameErrors":[{},{}],"level":"error","msg":"could not find a server certificate for the edge.api.address host [MY_HOST_PUBLIC_IP]","time":"2025-02-11T19:40:23.923Z"}
{"error":"could not get enrollment signer: could not determine enrollment signer: could not find a configured server certificate that matches hostname [MY_HOST_PUBLIC_IP] in root controller identity nor in xweb identities","file":"github.com/openziti/ziti/controller/api/responder.go:135","func":"github.com/openziti/ziti/controller/api.(*ResponderImpl).RespondWithError","level":"error","msg":"unhandled error returned to REST API","time":"2025-02-11T19:40:23.923Z","uri":"/edge/management/v1/edge-routers"}

The controller is accessible from my computer, I managed to login into ZAC aswell. So it's not really a port problem

BTW, really like your CLI, which one are you using ? Is it a themed Terminator or ?

That's the controller checking its own startup sanity. :ziggys_gone_crazy:

Basically, the controller (and the router) both make sure they have the necessary certificate for the addresses you told them to use, i.e., the "advertised host."

Here's the advertised host that the controller couldn't find a certificate for: edge.api.address (from the first log line).

It means you have a config like this:

edge:
  api:
    address: [MY_HOST_PUBLIC_IP]:1280

But, none of the controller's identities are for [MY_HOST_PUBLIC_IP], so the things you would have deployed later wouldn't have been able to verify this controller's cert.

It could be a bug in the configuration generator, but let's find out for sure. Please run this to probe the controller's client API advertised host to see which IP addresses and domain names the server's certificate is set to use for edge.api.address.

openssl s_client -connect [MY_HOST_PUBLIC_IP]:443 -alpn h2,http/1.1 <>/dev/null \
|& openssl x509 -noout -text \
| grep -A9 "Subject Alternative Name" \
| sed -E 's/,?\s+(DNS|IP|URI)/\n\t\t\1/g'

Thanks! The two terminals in the video are my Linux workstation and the VPS. My local workstation is using ZSH w/ Starship prompt (a tiny, fast, Rust binary), but you can use Starship wish other shells. I'm using fzf for history search, and zsh-autocompletions for command completions. I installed everything with Brew.

My starship.toml:

# Inserts a blank line between shell prompts
add_newline = true
command_timeout = 500

# Replace the "❯" symbol in the prompt with "➜"
#[character] # The name of the module we are configuring is "character"
#success_symbol = "[➜](bold green)" # The "success_symbol" segment is being set to "➜" with the color "bold green"

#[package]
#disabled = true
#
#[gcloud]
#disabled = true
#
#[python]
#disabled = true
#
#[cmake]
#disabled = true

format = """
$username\
$hostname\
$directory\
$git_branch\
$git_state\
$git_status\
$cmd_duration\
$line_break\
$python\
$character"""

[directory]
style = "blue"

[character]
success_symbol = "[❯](white)"
error_symbol = "[❯](red)"
vicmd_symbol = "[❮](green)"

[git_branch]
format = "[$branch]($style)"
style = "bright-black"

[git_status]
format = "[[(*$conflicted$untracked$modified$staged$renamed$deleted)](218)($ahead_behind$stashed)]($style)"
style = "cyan"
conflicted = "​"
untracked = "​"
modified = "​"
staged = "​"
renamed = "​"
deleted = "​"
stashed = "≡"

[git_state]
format = '\([$state( $progress_current/$progress_total)]($style)\) '
style = "bright-black"

[cmd_duration]
format = "[$duration]($style) "
style = "yellow"

[python]
format = "[$virtualenv]($style) "
style = "bright-black"

The DNS/IP subject alternative names (SANs) are clues, and it will help me to diagnose if you describe how you configured the controller.

Did you use the default method of docker run openziti/ziti-controller with some env vars to generate the controlller's config YAML?


Consider using an FQDN. I'm not 100% confident it's even possible to use an IPv4 with the configuration generator that's built in to the Docker image. I do know for sure it will work with an FQDN, and that's almost always necessary anyway for reliability because it's a real pain to change the controller's address. So much easier with DNS.

For testing or convenience, you can use a magic wildcard DNS. Just be aware you'd be trusting the DNS provider, e.g., anything.[MY_HOST_PUBLIC_IP].sslip.io.

This runs forever, I guees because port 443 isn't opened. I've tried with 1280 and got this error:

ubuntu@instance20250211180253:~$ openssl s_client -connect <PUBLIC_IP_ADDRESS>:1280 -alpn h2,http/1.1 <>/dev/null \
|& openssl x509 -noout -text \
| grep -A9 "Subject Alternative Name" \
| sed -E 's/,?\s+(DNS|IP|URI)/\n\t\t\1/g'
Could not read certificate from <stdin>
20406FA4D3E00000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:
Unable to load certificate

This is the exact docker command I'm using to run the controller:

sudo docker run --name openziti_controller --user 0 --rm --privileged --network openziti --platform linux/arm64 \
  -v /opt/openziti_controller:/ziti-controller \
  -e ZITI_AUTO_RENEW_CERTS=true \
  -e ZITI_BOOTSTRAP=true \
  -e ZITI_BOOTSTRAP_CONFIG=true \
  -e ZITI_BOOTSTRAP_DATABASE=true \
  -e ZITI_BOOTSTRAP_PKI=true \
  -e ZITI_CTRL_ADVERTISED_ADDRESS=<PUBLIC_IP_ADDRESS> \
  -e ZITI_CTRL_ADVERTISED_PORT=1280 \
  -e ZITI_PWD=super_secret \
  -p 1280:1280 \
  openziti/ziti-controller:1.3.3 run config.yml

I just tried with
ZITI_CTRL_ADVERTISED_ADDRESS=ctrl.<PUBLIC_IP_ADDRESS>.sslip.io and ran into the same error :confused:

The ip address is 79.72.31.69 if u wanna test something on your end

1 Like

I see the problem, and it's because the config generator (erroneously) assumes ZITI_CTRL_ADVERTISED_ADDRESS is an FQDN.

Not an FQDN, but incorrectly set as a DNS SAN:

DNS:79.72.31.69

Is using an FQDN an option for you? If not, how about the magic wildcard idea?

Ah, the configurator won't re-generate if a config already exists. You can reset everything from scratch with docker compose --volumes down (and delete /opt/openziti_controller on the Docker host) or set ZITI_BOOTSTRAP_CONFIG=force next time you run to re-generate.

What do you mean with the magic wildcard ?

I've been calling FQDNs like ctrl.<PUBLIC_IP_ADDRESS>.sslip.io "magic wildcard" because it's a wildcard DNS record that magically points to the IP in the name.

Yup removing the volume and using the FQDN worked. Is there any other alternative ? We can't just use the IPv4 Ip ?

1 Like

Ziti works with IPv4 but the configurator assumes incorrectly it's an FQDN. It's important to use an FQDN anyway since it's so difficult to change the controller's address. All the other Ziti stuff you'll deploy depends on knowing that address, so changing it is about the same amount of work as starting from scratch.

Alright, thanks alot !

1 Like

I know you set the main TCP port to 1280, but when I probed the cert I found it on 443. FYI just in case you have a mismatch.

Aight, is there any tutorial to setup identities/services/configs ?
I'm trying to do something very basic, I have a http server running on the same docker machine with the same network as the Ziti controller/edge router and I want to be able to access it without opening ports on Oracle Cloud

Check out the "simple service" wizard in the console. It does several things for you.

But first, make sure you have default router policies (#all/#all): edge router policy and service edge router policy.