Edit: I probably sound cranky, it’s 2am and I gotta let this go for the night. I am HAPPY to contribute some documentation corrections (if required).
Hi there, I looked through the docs and there are different ENVs between the docker version and the local install version, and it’s quite confusing trying to figure out what the correct values should be. How are we supposed to infer what the required variables are when there is no consistency between them?
Is there a place where ALL the environment values are listed for reference?
For instance, here are the ENVs for the docker compose setup. Note that some are commented out.
# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller
ZITI_CTRL_ADVERTISED_ADDRESS=ziti-edge-controller
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
#ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
#ZITI_CTRL_ADVERTISED_PORT=8440
# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080
# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080
# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
#ZITI_ROUTER_ADVERTISED_HOST=ziti-edge-router
#ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public
In the end, I am playing whack-a-mole with environment variables just to get a workind URL from the identity enrollment URL, which keeps coming up https://ziti-edge-controller:1280/edge/management/v1/identities
Quickstarts doc improvements are underway. @TheLumberjack is out of the office this week, but there may be places where he’d take you up on your offer to help once he returns.
Regarding docs for the address you were looking for:
I’ve created a GitHub issue to track progress on documenting those config vars.
Your immediate problem is that you want to specify the TCP port where your controller listens for edge clients, e.g., for enrollments, correct?
Clarifications that might help with reasoning about the config:
/edge/management/v1/ is the management API bound to the same TCP port of the web listener along with the client API /edge/client/v1/.
While it’s possible to configure the Ziti controller to bind these two APIs to separate web listeners, the quickstart binds them to the same port for simplicity’s sake.
This listener is configured with env vars: ${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT}, and ziti-edge-controller:1280 is the default value.
The management API is where the ziti edge CLI connects to create, read, update, and delete Ziti things, and the client API is where edge clients enroll and re-authenticate in the future.
I assume you’re attempting to get it running as a Docker Compose project. As I understand the quickstarts, they read those vars to compose the necessary configuration files on the first run, but they don’t update the configs if you change the vars.
Assuming that’s correct, you can always modify the configuration files to have the desired values. Alternatively, if you’re developing automation for one-shot deployments, you can start over with a fresh quickstart in Docker Compose by running docker compose down --volumes (destroys the named volume with the controller database).
If you need to modify the configuration of a running Ziti network in Docker Compose you use docker compose exec ziti-controller bash to gain a shell and change the file /persistent/ziti-controller.yaml. The doc that Dave linked above is the reference for this configuration file.
You must set the advertised address for the client API in property edge.api.address, and the web listener for the client API defined in the web[] list must have a matching address. Each web listener has a list of bindPoints and the address property of the one that binds edge-client must match edge.api.address.
ziti create config env which will output a list of export statements as well as commented descriptions and the current values of each environment variable so you can create your own custom env file.
I’ll take a look at the inconsistencies in the env vars, as there should not be any. All the quickstarts use the same core logic to determine the config output. The difference you see is mostly because they are different types of quickstarts. Our docker documentation doesn’t have details regarding how to use docker in a remotely-hosted environment. It was originally intended for users to spin docker up for testing. We later noticed there was a larger desire than expected to use docker for a remotely-hosted environment, and we’ve not made a good set of instructions for using that as of yet.
In the docker .env file, many of the variables are commented out because the defaults will allow you to spin up a docker env locally. The commented variables are optional specifications that commonly change given certain environments so they’re in there to give the user some customization options if they want that.
I do see how, in looking at these two lists of environment variables, is confusing, I’ll see what I can do to modify the doc and make that less confusing.
Thanks for all the kind replies. I really am sorry I was grumpy last night.
One thing I was speculating on was whether or not changing any of the bindings/ports/IPs/hostnames would then interfere with the docker network; now the services can find each other… but start changing these simple hostnames to a FQDN will the containers have to resolve outside the docker network? I hadn’t gotten that far yet and kind of doubt it having a chance to look at the list of variables so helpfully provided. My MVP is a running compose file and enroll a test system. Next iteration is a service.
There are a LOT of things going on under the hood… and I am determined to understand them.
Ok, kind of stuck still. I can change the advertised address. I looked for public addresses with ./ziti create config env -h | grep public and set the ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti.jptech.corp and ZITI_CTRL_ADVERTISED_ADDRESS=ziti.jptech.ninja. This resulted in all the certs as expected. But the other containers were looking for that public URL, and it wouldn’t resolve. So I changed the references in the docker-compose file to use the internal name for local communication… but this ended in TLS handshake errors.
So, for those running this via Docker Compose, how are you doing it? Can you share your docker-compose.yaml file?
# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti.jptech.ninja
ZITI_CTRL_ADVERTISED_ADDRESS=ziti.jptech.ninja
I'm loving that tenacity. It'll keep getting more convenient and systematic with high quality feedback like yours. Thanks.
Is it your main goal now to set up your Compose quickstart to host on a public IP or to handle a change of address on a running network or both?
It'll help us reason about changing addresses if we distill them into their Ziti principles.
Ziti has advertised addresses and bind addresses. For the most part, the name of the address configuration properties make it clear whether they define a bind address or an advertised address.
The advertisement is the address that other parts of the Ziti system use to connect to the listener. All the pieces of the Ziti system must be able to reach the advertised address (lookup the DNS name and route to the IP).
I'll stop there because I remember @gberl002 was working on specifics for setting the Compose quickstart's env vars to specify global DNS (resolves to a public IP).
When I’m using docker compose in a hosted environment, generally the minimum requirement is to modify/add the following to the .env file. This essentially is doing what you would have done in that first set of export statements for the “Host it anywhere” quickstart, we’re just applying/uncommenting the values for use in docker.
# Update this to the public DNS
ZITI_CTRL_ADVERTISED_ADDRESS=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
# Uncomment and update these appropriately, the defaults port values are what all doc generally uses
ZITI_CTRL_EDGE_IP_OVERRIDE=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
ZITI_CTRL_ADVERTISED_PORT=8440
# I believe there is a bug here that's on my plate to fix, I'll double check this
# but this should say _ADDRESS, the one in the env says "_HOST"
ZITI_ROUTER_ADVERTISED_ADDRESS=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
# Uncomment and update the following as needed
ZITI_ROUTER_PORT=8442
ZITI_ROUTER_IP_OVERRIDE=172.31.16.243
Here’s the fully edited .env file for review
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest
# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
ZITI_USER=admin
ZITI_PWD=
# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller
ZITI_CTRL_ADVERTISED_ADDRESS=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
ZITI_CTRL_EDGE_IP_OVERRIDE=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
ZITI_CTRL_ADVERTISED_PORT=8440
# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080
# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
ZITI_ROUTER_ADVERTISED_ADDRESS=ec2-3-143-246-113.us-east-2.compute.amazonaws.com
ZITI_ROUTER_PORT=8442
ZITI_ROUTER_IP_OVERRIDE=172.31.16.243
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public
Now, I can do a quick test and ping the controller’s edge
This should get you up to the point of having a functional, externally accessible hosted ziti network, let me know if this helps or if you’re still having issues.
I don't quite understand the second part. But my intention would be to run the minimum components on a host, like the controller, edgerouter and zac, if those are the three basic parts, in docker-compose. Then I would add additional resources as required. But start the bare bones, bone stock.
This also confirms the bug I was previously questioning, my DNS list isn’t showing the router advertised host I provided because that part of the code is looking for ZITI_ROUTER_ADVERTISED_HOST but most other places it’s looking for ZITI_ROUTER_ADVERTISED_ADDRESS. There was a mixup in the naming between the doc and a few parts in the code.
It would probably be beneficial to set both ZITI_ROUTER_ADVERTISED_HOST and ZITI_ROUTER_ADVERTISED_ADDRESS and set them both to the same value, your external DNS. At least until I get that bug fixed, which I’ll do first thing tomorrow as that’s a pretty bad one.
Something I didn’t mention before about my purpose… part of it is to make a repeatable process for creating an openZITI setup for each client. I had considered using one instance and separating it via policies. But I would like to limit the blast radius by keeping them separate. I have some clients in healthcare, some in finance, and some in law. So I am looking at a single cloud host with a light-weight orchestrated container service for each. Sometimes I only need to provide a VPN replacement at it’s simplest. But I would like to nail it down to a single repeatable configuration.
After I get the docker compose setup working, I can further orchestrate it with ansible. All upgrades just involve bouncing the orchestrator or changing the pinned container version, and this would result from a git commit on the left and a gitops action on the right and pipelines in between.
Anyway. This is what my overreaching goal is. If I am going to go to the trouble to ‘DIY’ this, I need to make the core service as dead simple and without fiddling as possible.
