Is quickstart working right now? (TAG 0.30.3-0ca6e6d7)

1

Hi there. I am trying to build a quickstart for some testing, and my current config is no longer working. I then pulled the raw files from the repo and set only the hostname (this is all internal, no public internet).

Here is my .env

$ cat .env
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest

# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
ZITI_USER=admin
ZITI_PWD=password

ZITI_INTERFACE=0.0.0.0

# controller name, address/port information
ZITI_CTRL_NAME=ziti-dev.jptech.corp
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-dev.jptech.corp
ZITI_CTRL_ADVERTISED_ADDRESS=ziti-dev.jptech.corp
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
ZITI_CTRL_ADVERTISED_PORT=8440

# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080

# router address/port information
ZITI_ROUTER_NAME=ziti-dev.jptech.corp
ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-dev.jptech.corp
ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
ZITI_ROUTER_LISTENER_BIND_PORT=8444
ZITI_ROUTER_ROLES=public

Here is the output of the logs.

$ docker compose -f simplified-docker-compose.yml logs
ziti-ziti-controller-1  | system has been initialized. starting the process.
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_BIN_DIR already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_BIN_ROOT already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_PORT already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_PORT already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_EDGE_IP_OVERRIDE already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_CTRL_NAME already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ENV_FILE already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_HOME already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_IMAGE already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_NETWORK already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_PWD already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_ENROLLMENT_DURATION already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_LISTENER_BIND_PORT already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_NAME already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_PORT already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_ROUTER_ROLES already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_SCRIPTS already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_SHARED already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_USER already set. using existing value
ziti-ziti-controller-1  | NOT OVERRIDING: env var ZITI_VERSION already set. using existing value
ziti-ziti-controller-1  |
ziti-ziti-controller-1  | adding /var/openziti/ziti-bin to the path
ziti-ziti-controller-1  | controller initialized. unsetting ZITI_USER/ZITI_PWD from env
ziti-ziti-controller-1  | [   0.009]    INFO ziti/ziti/controller.run: {go-version=[go1.20.7] nodeId=[ziti-dev.jptech.corp] os=[linux] arch=[amd64] build-date=[2023-09-01T21:03:45Z] revision=[c7a0a41867c4] version=[v0.30.3]} starting ziti-controller
ziti-ziti-controller-1   | [   0.009]    INFO storage/boltz.(*migrationManager).Migrate.func1: fabric datastore is up to date at version 5
ziti-ziti-controller-1   | [   0.010]    INFO fabric/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[100] idleTime=[30s] maxQueueSize=[100] poolType=[pool.router.messaging] minWorkers=[0]} starting goroutine pool
ziti-ziti-controller-1   | [   0.010]    INFO fabric/controller/network.(*Network).showOptions: network = {
ziti-ziti-controller-1   |   "CreateCircuitRetries": 2,
ziti-ziti-controller-1   |   "CycleSeconds": 60,
ziti-ziti-controller-1   |   "EnableLegacyLinkMgmt": true,
ziti-ziti-controller-1   |   "InitialLinkLatency": 65000000000,
ziti-ziti-controller-1   |   "IntervalAgeThreshold": 0,
ziti-ziti-controller-1   |   "MetricsReportInterval": 60000000000,
ziti-ziti-controller-1   |   "MinRouterCost": 10,
ziti-ziti-controller-1   |   "PendingLinkTimeout": 10000000000,
ziti-ziti-controller-1   |   "RouteTimeout": 10000000000,
ziti-ziti-controller-1   |   "RouterConnectChurnLimit": 60000000000,
ziti-ziti-controller-1   |   "RouterComm": {
ziti-ziti-controller-1   |     "QueueSize": 100,
ziti-ziti-console-1      | running ZAC
ziti-ziti-edge-router-1  | _ZITI_ROUTER_NAME set to: ziti-dev.jptech.corp
ziti-ziti-console-1      | ZAC will use this key for TLS: /persistent/pki/ziti-dev.jptech.corp-intermediate/keys/ziti-dev.jptech.corp-server.key
ziti-ziti-console-1      | ZAC will present this pem for TLS: /persistent/pki/ziti-dev.jptech.corp-intermediate/certs/ziti-dev.jptech.corp-server.cert
ziti-ziti-console-1      | emitting settings.json
ziti-ziti-console-1      | Loading Settings File From: /usr/src/app/../ziti/settings.json
ziti-ziti-console-1      | {
ziti-ziti-console-1      |   edgeControllers: [
ziti-ziti-console-1      |     {
ziti-ziti-console-1      |       name: 'ziti-dev.jptech.corp',
ziti-ziti-console-1      |       url: 'https://ziti-dev.jptech.corp:8441',
ziti-ziti-console-1      |       default: true
ziti-ziti-console-1      |     }
ziti-ziti-console-1      |   ],
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_BIN_DIR already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_BIN_ROOT already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_PORT already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_PORT already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_CTRL_NAME already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_ENV_FILE already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_HOME already set. using existing value
ziti-ziti-edge-router-1  | NOT OVERRIDING: env var ZITI_IMAGE already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_NETWORK already set. using existing value
ziti-ziti-controller-1   |     "MaxWorkers": 100
ziti-ziti-controller-1                 |   },
ziti-ziti-controller-1                 |   "Smart": {
ziti-ziti-console-1      |   editable: true,
ziti-ziti-console-1                    |   update: false,
ziti-ziti-console-1                    |   location: '../ziti',
ziti-ziti-console-1                    |   port: 1408,
ziti-ziti-console-1                    |   portTLS: 8443,
ziti-ziti-console-1                    |   logo: '',
ziti-ziti-console-1                    |   primary: '',
ziti-ziti-console-1                    |   secondary: '',
ziti-ziti-console-1                    |   allowPersonal: true,
ziti-ziti-console-1                    |   rejectUnauthorized: false,
ziti-ziti-controller-1                 |     "RerouteFraction": 0.02,
ziti-ziti-controller-1                 |     "RerouteCap": 4,
ziti-ziti-controller-1                 |     "MinCostDelta": 15
ziti-ziti-controller-1                 |   }
ziti-ziti-controller-1                 | }
ziti-ziti-controller-1                 | [   0.010]    INFO fabric/controller.(*Controller).showOptions: ctrl = {
ziti-ziti-controller-1                 |   "OutQueueSize": 4,
ziti-ziti-controller-1                 |   "MaxQueuedConnects": 1,
ziti-ziti-controller-1                 |   "MaxOutstandingConnects": 16,
ziti-ziti-controller-1                 |   "ConnectTimeout": 5000000000,
ziti-ziti-controller-1                 |   "DelayRxStart": false,
ziti-ziti-controller-1                 |   "WriteTimeout": 0,
ziti-ziti-controller-1                 |   "NewListener": null,
ziti-ziti-controller-1                 |   "AdvertiseAddress": null,
ziti-ziti-controller-1                 |   "RouterHeartbeatOptions": {
ziti-ziti-controller-1                 |     "sendInterval": 10000000000,
ziti-ziti-controller-1                 |     "checkInterval": 1000000000,
ziti-ziti-controller-1                 |     "closeUnresponsiveTimeout": 30000000000
ziti-ziti-controller-1                 |   },
ziti-ziti-controller-1                 |   "PeerHeartbeatOptions": {
ziti-ziti-controller-1                 |     "sendInterval": 10000000000,
ziti-ziti-controller-1                 |     "checkInterval": 1000000000,
ziti-ziti-console-1                    |   mail: { host: '', port: 25, secure: false, auth: { user: '', pass: '' } },
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_PWD already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_ADVERTISED_ADDRESS already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_ENROLLMENT_DURATION already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_LISTENER_BIND_PORT already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_NAME already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_PORT already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_ROUTER_ROLES already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_SCRIPTS already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_SHARED already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_USER already set. using existing value
ziti-ziti-edge-router-1                | NOT OVERRIDING: env var ZITI_VERSION already set. using existing value
ziti-ziti-edge-router-1                |
ziti-ziti-edge-router-1                | adding /var/openziti/ziti-bin to the path
ziti-ziti-edge-router-1                | ZITI_ROUTER_NAME set to: ziti-dev.jptech.corp
ziti-ziti-edge-router-1                |  Found existing config file /persistent/ziti-dev.jptech.corp.yaml, not creating a new config.
ziti-ziti-edge-router-1                | [   0.008] WARNING fabric/router.LoadConfig: invalid [healthChecks] stanza
ziti-ziti-edge-router-1                | [   0.008]    INFO ziti/ziti/router.run: {revision=[c7a0a41867c4] arch=[amd64] configFile=[/persistent/ziti-dev.jptech.corp.yaml] routerId=[ziti-dev.jptech.corp] build-date=[2023-09-01T21:03:45Z] version=[v0.30.3] go-version=[go1.20.7] os=[linux]} starting ziti-router
ziti-ziti-edge-router-1                | [   0.008]    INFO fabric/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[32] idleTime=[30s] poolType=[pool.link.dialer] maxQueueSize=[1000] minWorkers=[0]} starting goroutine pool
ziti-ziti-edge-router-1                | [   0.008]    INFO fabric/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.route.handler] minWorkers=[0] maxWorkers=[128] idleTime=[30s] maxQueueSize=[1000]} starting goroutine pool
ziti-ziti-edge-router-1                | [   0.009] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
ziti-ziti-edge-router-1                | [   0.009]    INFO fabric/router/forwarder.(*Scanner).run: started
ziti-ziti-edge-router-1                | [   0.009]    INFO fabric/router/forwarder.(*Faulter).run: started
ziti-ziti-console-1                    |   from: '',
ziti-ziti-console-1                    |   to: ''
ziti-ziti-controller-init-container-1  |
ziti-ziti-console-1                    | }
ziti-ziti-controller-init-container-1  | *****************************************************************
ziti-ziti-controller-init-container-1  |  docker-compose init file has been detected, the initialization
ziti-ziti-controller-init-container-1  |  of the docker-compose environment has already happened. If you
ziti-ziti-controller-init-container-1  |  wish to allow this volume to be re-initialized, delete the file
ziti-ziti-controller-init-container-1  |  located at /persistent/access-control.init
ziti-ziti-controller-init-container-1  | *****************************************************************
ziti-ziti-controller-init-container-1  |
ziti-ziti-console-1                    | TLS initialized on port: 8443
ziti-ziti-console-1                    | Ziti Admin Console is now listening on port 1408
ziti-ziti-edge-router-1                | [   0.009]   PANIC ziti/ziti/router.run: {error=[required section [edge.csr] not found]} error registering edge in framework
ziti-ziti-edge-router-1                | panic: (*logrus.Entry) 0xc0000f5420
ziti-ziti-edge-router-1                |
ziti-ziti-edge-router-1                | goroutine 1 [running]:
ziti-ziti-edge-router-1                | github.com/sirupsen/logrus.(*Entry).log(0xc0000f5340, 0x0, {0xc0005e2c00, 0x23})
ziti-ziti-edge-router-1                |        github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
ziti-ziti-edge-router-1                | github.com/sirupsen/logrus.(*Entry).Log(0xc0000f5340, 0x0, {0xc0005d10e0?, 0x320ac60?, 0xc0005f8e10?})
ziti-ziti-edge-router-1                |        github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
ziti-ziti-edge-router-1                | github.com/sirupsen/logrus.(*Entry).Panic(...)
ziti-ziti-edge-router-1                |        github.com/sirupsen/logrus@v1.9.3/entry.go:342
ziti-ziti-edge-router-1                | github.com/openziti/ziti/ziti/router.run(0xc0009e3800?, {0xc000a9dc40, 0x1, 0x1?})
ziti-ziti-edge-router-1                |        github.com/openziti/ziti/ziti/router/run.go:81 +0xa94
ziti-ziti-edge-router-1                | github.com/spf13/cobra.(*Command).execute(0xc0009e3800, {0xc000a9dc10, 0x1, 0x1})
ziti-ziti-edge-router-1                |        github.com/spf13/cobra@v1.7.0/command.go:944 +0x847
ziti-ziti-edge-router-1                | github.com/spf13/cobra.(*Command).ExecuteC(0x5285aa0)
ziti-ziti-edge-router-1                |        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
ziti-ziti-edge-router-1                | github.com/spf13/cobra.(*Command).Execute(...)
ziti-ziti-edge-router-1                |        github.com/spf13/cobra@v1.7.0/command.go:992
ziti-ziti-edge-router-1                | github.com/openziti/ziti/ziti/cmd.Execute()
ziti-ziti-edge-router-1                |        github.com/openziti/ziti/ziti/cmd/cmd.go:79 +0x25
ziti-ziti-edge-router-1                | main.main()
ziti-ziti-edge-router-1                |        github.com/openziti/ziti/ziti/main.go:51 +0x17
ziti-ziti-controller-1                 |     "closeUnresponsiveTimeout": 30000000000
ziti-ziti-controller-1                 |   }
ziti-ziti-controller-1                 | }
ziti-ziti-controller-1                 | [   0.366]    INFO edge/controller/server.NewController: edge controller instance id: cln2jk5rg00000bpnctxp4mfb
ziti-ziti-controller-1                 | [   0.366]    INFO edge/controller/server.(*Controller).Initialize: initializing edge
ziti-ziti-controller-1                 | [   0.369]    INFO storage/boltz.(*migrationManager).Migrate.func1: edge datastore is up to date at version 33
ziti-ziti-controller-1                 | [   0.371]    INFO edge/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
ziti-ziti-controller-1                 | [   0.371]    INFO edge/controller/server.(*Controller).Run: starting edge
ziti-ziti-controller-1                 | [   0.371]    INFO fabric/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[16] idleTime=[10s] minWorkers=[1] maxQueueSize=[1] poolType=[pool.listener.ctrl]} starting goroutine pool
ziti-ziti-controller-1                 | [   0.371]    INFO channel/v2.(*UnderlayDispatcher).Run: started
ziti-ziti-controller-1                 | [   0.374]    INFO edge/controller/server.(*Controller).checkEdgeInitialized: edge initialized
ziti-ziti-controller-1                 | [   0.382]    INFO fabric/controller/network.(*Network).Run: started
ziti-ziti-controller-1                 | [   0.464]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:8441 for server client-management with APIs: [edge-management edge-client fabric]
2

I'll give it a go and let you know. I know @qrkourier was touching the cloudfront script that supports get.openziti.io today. It was working earlier, but I can see your edge router isn't starting properly. Everything else seems ok.

I'll followup in a bit...

3

I ran it a couple of times on my VM and I'm sad to report, it worked fine for me. The error you show above is a bit strange:

required section [edge.csr] not found]

it's reminiscent of the issue you had before where it makes me think the file is empty. It feels/looks like that problem again to me. That one I couldn't reproduce. :ziggy_sad_closeup:

4

Good grief... I am like the bermuda triangle, LOL.

Mind sharing your .env file so I can compare it?

5

i just got the default and ran it... tried the latest, tried 0.30.3 too... I'd start with that just to see if it works for you? TBH, I don't think it'll matter anyway. It's like the router's config file isn't complete when the process starts. I even copied yours, changed the advertised address and it works fine too

cat .env
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest
#ZITI_VERSION=0.30.3

# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
ZITI_USER=admin
ZITI_PWD=

ZITI_INTERFACE=0.0.0.0

# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller
ZITI_CTRL_ADVERTISED_ADDRESS=ziti-controller
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
#ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
#ZITI_CTRL_ADVERTISED_PORT=8440

# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080

# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
#ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-edge-router
#ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public
6

I think I figured it out. If I rename the router to the same name as the controller it breaks. Interestingly, this was not a problem on the 0.29.? version I am running in pseudo-production.

$ diff .env .env.broke
58c58
< ZITI_ROUTER_NAME=ziti-dev-router.jptech.corp
---
> ZITI_ROUTER_NAME=ziti-dev.jptech.corp
7

oh that makes a ton of sense. the 'name' might be used to output the yaml file in the persistent folder, that would mean it gets overwritten by 'one or the other'...

@gberl002 can you confirm what happens when you try this with a simplified compose?

Finally, it sounds like we might understand why I could never reproduce it