Edge-router (containerized) unable to connect to controller

Dear all,

I have the containerized controller and ZAC up and running and try to get the containerized router connected to the controller, which failes.

ZITI_ROUTER_TYPE=edge
ZITI_CTRL_ADVERTISED_PORT=1280
ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-fabric-router-01.rooibos.siemens.cloud

openziti-router-ziti-router-1 | {"endpoint":{"tls:ziti-ctrl.rooibos.siemens.cloud:1280":{}},"file":"github.com/openziti/ziti/router/env/ctrls.go:95","func":"github.com/openziti/ziti/router/env.(*networkControllers).UpdateControllerEndpoints","level":"info","msg":"adding new ctrl endpoint","time":"2025-01-15T17:56:56.002Z"}

openziti-router-ziti-router-1 | {"endpoint":"tls:ziti-ctrl.rooibos.siemens.cloud:1280","file":"github.com/openziti/ziti/router/env/ctrls.go:134","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff","level":"info","msg":"starting connection attempts","time":"2025-01-15T17:56:56.002Z"}

openziti-router-ziti-router-1 | {"endpoint":"tls:ziti-ctrl.rooibos.siemens.cloud:1280","error":"error connecting ctrl (EOF)","file":"github.com/openziti/ziti/router/env/ctrls.go:129","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2","level":"error","msg":"unable to connect controller","time":"2025-01-15T17:56:56.079Z"}

openziti-router-ziti-router-1 | {"endpoint":"tls:ziti-ctrl.rooibos.siemens.cloud:1280","error":"error connecting ctrl (EOF)","file":"github.com/openziti/ziti/router/env/ctrls.go:129","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func2","level":"error","msg":"unable to connect controller","time":"2025-01-15T17:56:56.209Z"}

I starte a curl in the router container to the controller which shows me an empty reply, so the connectivity seams to be ok.

[ziggy@4c82cd9b148a ~]$ curl ziti-ctrl.rooibos.siemens.cloud:1280
curl: (52) Empty reply from server

any hint?

Generally speaking this is "https" vs "http". Since you didn't supply https, it's using http... I'm pretty sure that's the problem here...

you can replicate this just by running ziti edge quickstart

cd@192.168.253.239:sg4: ~
$ curl localhost:1280
curl: (52) Empty reply from server
cd@192.168.253.239:sg4: ~
$ curl -ks https://localhost:1280
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://sg4:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://sg4:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://sg4:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-10-02T12:59:41Z","capabilities":[],"revision":"0eec47ce3c80","runtimeVersion":"go1.23.1","version":"v1.1.15"},"meta":{}}
cd@192.168.253.239:sg4: ~
$

easy to do, i've done it :slight_smile:

As for the edge-router not being able to connect to the controller (i see i didn't answer that sorry) I would guess that the advertised address of the controller is not routable from the router container/vm.

my curl reponse, started inside the router conatiner

[ziggy@4c82cd9b148a ~]$ curl -ks https://ziti-ctrl.rooibos.siemens.cloud:1280
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://ziti-ctrl.rooibos.siemens.cloud:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://ziti-ctrl.rooibos.siemens.cloud:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://ziti-ctrl.rooibos.siemens.cloud:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-10-02T12:59:41Z","capabilities":[],"revision":"0eec47ce3c80","runtimeVersion":"go1.23.1","version":"v1.1.15"},"meta":{}}

seems to be routed to or ?

It seems that TCP port is reachable from the Docker router. It makes me wonder if there's a different problem with the router-to-controller connection, besides the connectivity itself. Perhaps a certificate or protocol problem.

You can add in your router's Docker compose.yml (link to example) like command: run config.yml --verbose to set DEBUG log level. That may shed light on the nature of the connection problem.

Please surround log pastes with ``` fences so they're easier to read. Thanks!

dear all, sorry for the long delay, coming back now. I am running in this kind of problem only when I modify the ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=zitiede-ctrl.ddns.net to a FQDN. When I am running with the default settings all is fine. Any hint for me?

user@overlay:~/.config/ziti/certs$ ziti edge quickstart
emitting a minimal PKI
Success
Using CA name:  root-ca
Success
Using CA name:  intermediate-ca
Success
Using CA name:  intermediate-ca
Success
[   3.227] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://09de6b93c50bd0807295d7deac66cdbe5bdd8cdc], it is recommended that a trust domain is specified in configuration via URI SANs or the                      'trustDomain' field
[   3.227] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://09de6b93c50bd0807295d7deac66cdbe5bdd8cdc], it is recommended that if network components have enrolled that the generated trust dom                     ain be added to the configuration field 'additionalTrustDomains' array when configuring a explicit trust domain
[   3.233]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: Migrated edge datastore from 0 to 37
[   3.233]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: edge datastore is up to date at version 37
[   3.991]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] maxQueueSize=[100] poolType=[pool.router.messaging] minWorkers=[0] maxWorkers=[100]} starting goroutine pool
[   3.991]    INFO ziti/controller/network.(*Network).showOptions: network = {
  "CreateCircuitRetries": 2,
  "CycleSeconds": 60,
  "EnableLegacyLinkMgmt": false,
  "InitialLinkLatency": 65000000000,
  "IntervalAgeThreshold": 0,
  "MetricsReportInterval": 60000000000,
  "MinRouterCost": 10,
  "PendingLinkTimeout": 10000000000,
  "RouteTimeout": 10000000000,
  "RouterConnectChurnLimit": 60000000000,
  "RouterComm": {
    "QueueSize": 100,
    "MaxWorkers": 100
  },
  "Smart": {
    "RerouteFraction": 0.02,
    "RerouteCap": 4,
    "MinCostDelta": 15
  }
}
[   3.991]    INFO ziti/controller/webapis.NewFabricManagementApiFactory: initializing management api factory with 0 xmgmt instances
[   3.991]    INFO ziti/controller.(*Controller).showOptions: ctrl = {
  "OutQueueSize": 4,
  "MaxQueuedConnects": 1,
  "MaxOutstandingConnects": 16,
  "ConnectTimeout": 5000000000,
  "DelayRxStart": false,
  "WriteTimeout": 0,
  "MessageStrategy": null,
  "NewListener": null,
  "AdvertiseAddress": {},
  "RouterHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  },
  "PeerHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  }
}
[   3.991]    INFO ziti/controller/server.NewController: edge controller instance id: cm6kxua910000lsmlh20lc7xv
[   3.992]    INFO ziti/controller.(*Controller).RegisterXmgmt: adding xmgmt *server.submgmt, enabled? true
[   3.992]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   3.994]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {frequency=[5s] sessionTimeout=[30m0s]} session enforcer configured
[   4.004]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: shutting down...
[   4.004]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: stopped
[   4.004]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: shutting down...
[   4.004]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: stopped
[   4.004]    INFO ziti/controller/server.(*Controller).Shutdown: shutdown complete
[   4.004]    INFO ziti/controller/subcmd.NewEdgeInitializeCmd.func2: Ziti Edge initialization complete
Controller running... Configuring and starting Router...
[   4.023] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://09de6b93c50bd0807295d7deac66cdbe5bdd8cdc], it is recommended that a trust domain is specified in configuration via URI SANs or the                      'trustDomain' field
[   4.023] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://09de6b93c50bd0807295d7deac66cdbe5bdd8cdc], it is recommended that if network components have enrolled that the generated trust dom                     ain be added to the configuration field 'additionalTrustDomains' array when configuring a explicit trust domain
[   4.024]    INFO ziti/ziti/controller.run: {build-date=[2024-10-02T12:59:41Z] go-version=[go1.23.1] os=[linux] revision=[0eec47ce3c80] nodeId=[client] arch=[amd64] version=[v1.1.15]} starting ziti-controller
[   4.029]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: edge datastore is up to date at version 37
[   4.789]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[100] maxWorkers=[100] idleTime=[30s] poolType=[pool.router.messaging] minWorkers=[0]} starting goroutine pool
[   4.789]    INFO ziti/controller/network.(*Network).showOptions: network = {
  "CreateCircuitRetries": 2,
  "CycleSeconds": 60,
  "EnableLegacyLinkMgmt": false,
  "InitialLinkLatency": 65000000000,
  "IntervalAgeThreshold": 0,
  "MetricsReportInterval": 60000000000,
  "MinRouterCost": 10,
  "PendingLinkTimeout": 10000000000,
  "RouteTimeout": 10000000000,
  "RouterConnectChurnLimit": 60000000000,
  "RouterComm": {
    "QueueSize": 100,
    "MaxWorkers": 100
  },
  "Smart": {
    "RerouteFraction": 0.02,
    "RerouteCap": 4,
    "MinCostDelta": 15
  }
}
[   4.789]    INFO ziti/controller/webapis.NewFabricManagementApiFactory: initializing management api factory with 0 xmgmt instances
[   4.789] WARNING ziti/controller/webapis.OverrideRequestWrapper: requestWrapper overridden more than once
[   4.789]    INFO ziti/controller.(*Controller).showOptions: ctrl = {
  "OutQueueSize": 4,
  "MaxQueuedConnects": 1,
  "MaxOutstandingConnects": 16,
  "ConnectTimeout": 5000000000,
  "DelayRxStart": false,
  "WriteTimeout": 0,
  "MessageStrategy": null,
  "NewListener": null,
  "AdvertiseAddress": {},
  "RouterHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  },
  "PeerHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  }
}
[   4.789]    INFO ziti/controller/server.NewController: edge controller instance id: cm6kxuav70001lsmlju1yzkrx
[   4.790]    INFO ziti/controller.(*Controller).RegisterXmgmt: adding xmgmt *server.submgmt, enabled? true
[   4.790]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   4.792]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {frequency=[5s] sessionTimeout=[30m0s]} session enforcer configured
[   4.792]    INFO ziti/controller/server.(*Controller).Run: starting edge
[   4.792]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] idleTime=[10s] maxQueueSize=[1] maxWorkers=[16] poolType=[pool.listener.ctrl]} starting goroutine pool
[   4.793]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[   4.793]    INFO ziti/controller/server.(*Controller).checkEdgeInitialized: edge initialized
[   4.848]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51664] error=[tls: client didn't provide a certificate]} handshake failed
[   4.931]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51670] error=[tls: client didn't provide a certificate]} handshake failed
[   5.005]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:1280 for server client-management with APIs: [edge-management edge-client fabric]
[   5.006]    INFO ziti/controller/network.(*Network).Run: started
[   5.015]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51676] error=[tls: client didn't provide a certificate]} handshake failed
[   5.099]    INFO ziti/ziti/cmd/edge.(*QuickstartOpts).run: Controller online. Continuing...
[   5.154]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51698] error=[remote error: tls: bad certificate]} handshake failed
Untrusted certificate authority retrieved from server
Verified that server supplied certificates are trusted by server
Server supplied 2 certificates
Server certificate chain written to /home/user/.config/ziti/certs/zitiede-ctrl.ddns.net
RESTY 2025/01/31 15:47:49 ERROR Get "https://zitiede-ctrl.ddns.net:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for localhost, ziti-controller, not zitiede-ctrl.ddns.net, Attempt 1
[   5.254]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51732] error=[remote error: tls: bad certificate]} handshake failed
RESTY 2025/01/31 15:47:49 ERROR Get "https://zitiede-ctrl.ddns.net:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for localhost, ziti-controller, not zitiede-ctrl.ddns.net, Attempt 2
[   5.385]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51742] error=[remote error: tls: bad certificate]} handshake failed
RESTY 2025/01/31 15:47:49 ERROR Get "https://zitiede-ctrl.ddns.net:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for localhost, ziti-controller, not zitiede-ctrl.ddns.net, Attempt 3
[   5.536]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.100.80:51744] error=[remote error: tls: bad certificate]} handshake failed
RESTY 2025/01/31 15:47:49 ERROR Get "https://zitiede-ctrl.ddns.net:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for localhost, ziti-controller, not zitiede-ctrl.ddns.net, Attempt 4
[   5.787]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: bad certificate] remote=[192.168.100.80:51752]} handshake failed
RESTY 2025/01/31 15:47:50 ERROR Get "https://zitiede-ctrl.ddns.net:1280/edge/client/v1/version": tls: failed to verify certificate: x509: certificate is valid for localhost, ziti-controller, not zitiede-ctrl.ddns.net, Attempt 5

hi @Reinhard

When you run ziti edge quickstart you can supply a few fields. I can surmise from your post that you're trying to access the controller at https://zitiede-ctrl.ddns.net but when ziti edge quickstart runs, it made a pki that was valid for localhost and ziti-controller.... but not zitiede-ctrl.ddns.net.

You can fix this by simply adding: --ctrl-address zitiede-ctrl.ddns.net and --router-address zitiede-ctrl.ddns.net to your command.

ziti edge quickstart \
    --ctrl-address zitiede-ctrl.ddns.net \
    --router-address zitiede-ctrl.ddns.net

Also the --help flag provides useful details:

Usage:
  ziti edge quickstart [flags]

Flags:
      --ctrl-address string     sets the advertised address for the control plane and API. current: sg4
      --ctrl-port int16         sets the port to use for the control plane and API. current: 1280 (default 1280)
  -h, --help                    help for quickstart
      --home string             permanent directory
      --no-router               specifies the quickstart should not start a router
  -p, --password string         admin password, default: admin
      --router-address string   sets the advertised address for the integrated router. current: sg4
      --router-port int16       sets the port to use for the integrated router. current: 3022 (default 3022)
  -u, --username string         admin username, default: admin
      --verbose                 Show additional output.
1 Like

stepping over to the qickstart docker-compose scenario, all is working fine If I don't reconfigure the default environment values. but if I set

ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=zitiedge-ctrl.ddns.net
ZITI_CTRL_ADVERTISED_ADDRESS=ziti-ctrl.ddns.net

I get

user@overlay:~/openziti$ docker logs openziti-ziti-console-1
waiting for server key to exist...
waiting for server key to exist...

and the docker logs of controller shows at the end of the log info TLS certificate errors

user@overlay:~$ docker logs openziti-ziti-controller-1
system has not been initialized. initializing...
Populating environment variables
ZITI_NETWORK overridden: ziti
ZITI_HOME overridden: /persistent
ZITI_USER overridden: admin
ZITI_PWD overridden: xxxxxxxxxxxxxx
ZITI_BIN_DIR overridden: /var/openziti/ziti-bin
ZITI_CTRL_NAME overridden: ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_PORT overridden: 1280
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS overridden: zitiedge-ctrl.ddns.net
ZITI_CTRL_ADVERTISED_ADDRESS overridden: ziti-controller
ZITI_CTRL_ADVERTISED_PORT overridden: 6262
ZITI_HOME overridden: /persistent
ZITI_ENV_FILE overridden: /persistent/ziti.env
Your OpenZiti environment has been set up successfully.

A file with all pertinent environment values was created here: /persistent/ziti.env

NOT OVERRIDING: env var ZITI_ARCH already set. using existing value
NOT OVERRIDING: env var ZITI_BINARIES_FILE already set. using existing value
NOT OVERRIDING: env var ZITI_BINARIES_VERSION already set. using existing value
NOT OVERRIDING: env var ZITI_BIN_DIR already set. using existing value
NOT OVERRIDING: env var ZITI_BIN_ROOT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_IP_OVERRIDE already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ENV_FILE already set. using existing value
NOT OVERRIDING: env var ZITI_HOME already set. using existing value
NOT OVERRIDING: env var ZITI_IMAGE already set. using existing value
NOT OVERRIDING: env var ZITI_INTERFACE already set. using existing value
NOT OVERRIDING: env var ZITI_NETWORK already set. using existing value
NOT OVERRIDING: env var ZITI_OSTYPE already set. using existing value
NOT OVERRIDING: env var ZITI_PKI already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_CA already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_CERT already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_EDGE_ROOTCA_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_INTERMEDIATE_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_KEY already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_ROOTCA_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_CTRL_SERVER_CERT already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_EDGE_CA already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_EDGE_CERT already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_EDGE_KEY already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_EDGE_SERVER_CERT already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_SIGNER_CERT already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_SIGNER_CERT_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_SIGNER_INTERMEDIATE_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_SIGNER_KEY already set. using existing value
NOT OVERRIDING: env var ZITI_PKI_SIGNER_ROOTCA_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_PWD already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_LISTENER_BIND_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_SCRIPTS already set. using existing value
NOT OVERRIDING: env var ZITI_SHARED already set. using existing value
NOT OVERRIDING: env var ZITI_USER already set. using existing value
NOT OVERRIDING: env var ZITI_VERSION already set. using existing value

adding /var/openziti/ziti-bin to the path
Populating environment variables
ZITI_HOME overridden: /persistent
ZITI_USER overridden: admin
ZITI_PWD overridden: Ro1b00s1!
ZITI_PKI overridden: /persistent/pki
ZITI_PKI_SIGNER_CERT_NAME overridden: ziti-signing
ZITI_PKI_SIGNER_ROOTCA_NAME overridden: ziti-signing-root-ca
ZITI_PKI_SIGNER_INTERMEDIATE_NAME overridden: ziti-signing-intermediate
ZITI_PKI_SIGNER_CERT overridden: /persistent/pki/signing.pem
ZITI_PKI_SIGNER_KEY overridden: /persistent/pki/ziti-signing-intermediate/keys/ziti-signing-intermediate.key
ZITI_BIN_DIR overridden: /var/openziti/ziti-bin
ZITI_CTRL_NAME overridden: ziti-controller
ZITI_CTRL_EDGE_NAME overridden: ziti-edge-controller
ZITI_CTRL_EDGE_ADVERTISED_PORT overridden: 1280
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS overridden: zitiedge-ctrl.ddns.net
ZITI_CTRL_ADVERTISED_ADDRESS overridden: ziti-controller
ZITI_CTRL_ADVERTISED_PORT overridden: 6262
ZITI_PKI_CTRL_ROOTCA_NAME overridden: ziti-controller-root-ca
ZITI_PKI_CTRL_INTERMEDIATE_NAME overridden: ziti-controller-intermediate
ZITI_PKI_CTRL_EDGE_ROOTCA_NAME overridden: ziti-edge-controller-root-ca
ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME overridden: ziti-edge-controller-intermediate
ZITI_PKI_CTRL_SERVER_CERT overridden: /persistent/pki/ziti-controller-intermediate/certs/ziti-controller-server.chain.pem
ZITI_PKI_CTRL_KEY overridden: /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-server.key
ZITI_PKI_CTRL_CA overridden: /persistent/pki/cas.pem
ZITI_PKI_CTRL_CERT overridden: /persistent/pki/ziti-controller-intermediate/certs/ziti-controller-client.chain.pem
ZITI_PKI_EDGE_CERT overridden: /persistent/pki/ziti-edge-controller-intermediate/certs/zitiedge-ctrl.ddns.net-client.chain.pem
ZITI_PKI_EDGE_SERVER_CERT overridden: /persistent/pki/ziti-edge-controller-intermediate/certs/zitiedge-ctrl.ddns.net-server.chain.pem
ZITI_PKI_EDGE_KEY overridden: /persistent/pki/ziti-edge-controller-intermediate/keys/zitiedge-ctrl.ddns.net-server.key
ZITI_PKI_EDGE_CA overridden: /persistent/pki/ziti-edge-controller-root-ca/certs/ziti-edge-controller-root-ca.cert
ZITI_ROUTER_NAME overridden: ziti-edge-router
ZITI_ROUTER_PORT overridden: 3022
ZITI_ROUTER_LISTENER_BIND_PORT overridden: 10080
ZITI_HOME overridden: /persistent
ZITI_ENV_FILE overridden: /persistent/ziti.env
Your OpenZiti environment has been set up successfully.

A file with all pertinent environment values was created here: /persistent/ziti.env

Generating PKI
Creating CA: ziti-controller-root-ca
Success

Creating CA: ziti-edge-controller-root-ca
Success

Creating CA: ziti-signing-root-ca
Success

Creating intermediate: ziti-controller-root-ca ziti-controller-intermediate 1
Using CA name:  ziti-controller-root-ca
Success

Creating intermediate: ziti-edge-controller-root-ca ziti-edge-controller-intermediate 1
Using CA name:  ziti-edge-controller-root-ca
Success

Creating intermediate: ziti-signing-root-ca ziti-signing-intermediate_grandparent_intermediate 2
Using CA name:  ziti-signing-root-ca
Success

Creating intermediate: ziti-signing-intermediate_grandparent_intermediate ziti-signing-intermediate 1
Using CA name:  ziti-signing-intermediate_grandparent_intermediate
Success


Creating server cert from ca: ziti-controller-intermediate for localhost,ziti,ziti-controller / 127.0.0.1
Using CA name:  ziti-controller-intermediate
Success
Creating client cert from ca: ziti-controller-intermediate for localhost,ziti,ziti-controller
Using CA name:  ziti-controller-intermediate
Success

Creating server cert from ca: ziti-edge-controller-intermediate for localhost,ziti,zitiedge-ctrl.ddns.net / 127.0.0.1
Using CA name:  ziti-edge-controller-intermediate
Success
Creating client cert from ca: ziti-edge-controller-intermediate for localhost,ziti,zitiedge-ctrl.ddns.net
Using CA name:  ziti-edge-controller-intermediate
Success

PKI generated successfully


/persistent/ziti-edge-controller.yaml doesn't exist. Generating config file

adding controller root CA to ca bundle: /persistent/pki/ziti-controller-root-ca/certs/ziti-controller-root-ca.cert
adding signing root CA to ZITI_PKI_CTRL_CA: /persistent/pki/cas.pem
wrote CA file to: /persistent/pki/cas.pem
adding parent intermediate CA to ZITI_PKI_SIGNER_CERT: /persistent/pki/signing.pem
adding grandparent intermediate CA to ZITI_PKI_SIGNER_CERT: /persistent/pki/signing.pem
wrote signer cert file to: /persistent/pki/signing.pem
Controller configuration file written to: /persistent/ziti-controller.yaml
[   0.011] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://3c3ee359a633ab54c467e5cda8fe24fd93cf2a8f], it is recommended that a trust domain is specified in configuration via URI SANs or the 'trustDomain' field
[   0.011] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://3c3ee359a633ab54c467e5cda8fe24fd93cf2a8f], it is recommended that if network components have enrolled that the generated trust domain be added to the configuration field 'additionalTrustDomains' array when configuring a explicit trust domain
[   0.018]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: Migrated edge datastore from 0 to 37
[   0.018]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: edge datastore is up to date at version 37
[   0.823]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[100] idleTime=[30s] maxWorkers=[100] poolType=[pool.router.messaging] minWorkers=[0]} starting goroutine pool
[   0.823]    INFO ziti/controller/network.(*Network).showOptions: network = {
  "CreateCircuitRetries": 2,
  "CycleSeconds": 60,
  "EnableLegacyLinkMgmt": false,
  "InitialLinkLatency": 65000000000,
  "IntervalAgeThreshold": 0,
  "MetricsReportInterval": 60000000000,
  "MinRouterCost": 10,
  "PendingLinkTimeout": 10000000000,
  "RouteTimeout": 10000000000,
  "RouterConnectChurnLimit": 60000000000,
  "RouterComm": {
    "QueueSize": 100,
    "MaxWorkers": 100
  },
  "Smart": {
    "RerouteFraction": 0.02,
    "RerouteCap": 4,
    "MinCostDelta": 15
  }
}
[   0.823]    INFO ziti/controller/webapis.NewFabricManagementApiFactory: initializing management api factory with 0 xmgmt instances
[   0.823]    INFO ziti/controller.(*Controller).showOptions: ctrl = {
  "OutQueueSize": 4,
  "MaxQueuedConnects": 1,
  "MaxOutstandingConnects": 16,
  "ConnectTimeout": 5000000000,
  "DelayRxStart": false,
  "WriteTimeout": 0,
  "MessageStrategy": null,
  "NewListener": null,
  "AdvertiseAddress": {},
  "RouterHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  },
  "PeerHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  }
}
[   0.823]    INFO ziti/controller/server.NewController: edge controller instance id: cm6kyt2430000t3o53nlrn8ce
[   0.823]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   0.823]    INFO ziti/controller/sync_strats.(*InstantStrategy).Initialize: {logSize=[10000] listenerBufferSizes=[1000]} initialized controller router data model
[   0.825]    INFO ziti/controller/sync_strats.(*InstantStrategy).BuildAll.func1: {index=[0]} initialized router data model from db
[   0.825]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
[   0.836]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: shutting down...
[   0.836]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: stopped
[   0.836]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: shutting down...
[   0.837]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: stopped
[   0.837]    INFO ziti/controller/server.(*Controller).Shutdown: shutdown complete
[   0.837]    INFO ziti/controller/subcmd.NewEdgeInitializeCmd.func2: Ziti Edge initialization complete
controller initialized. unsetting ZITI_USER/ZITI_PWD from env
[   0.009] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://3c3ee359a633ab54c467e5cda8fe24fd93cf2a8f], it is recommended that a trust domain is specified in configuration via URI SANs or the 'trustDomain' field
[   0.009] WARNING ziti/controller/config.LoadConfig: this environment is using a default generated trust domain [spiffe://3c3ee359a633ab54c467e5cda8fe24fd93cf2a8f], it is recommended that if network components have enrolled that the generated trust domain be added to the configuration field 'additionalTrustDomains' array when configuring a explicit trust domain
[   0.010]    INFO ziti/ziti/controller.run: {revision=[94922ed93595] nodeId=[ziti-controller] build-date=[2025-01-25T14:43:16Z] version=[v1.3.2] go-version=[go1.23.4] os=[linux] arch=[amd64]} starting ziti-controller
[   0.013]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: edge datastore is up to date at version 37
[   0.861]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.router.messaging] maxWorkers=[100] idleTime=[30s] maxQueueSize=[100] minWorkers=[0]} starting goroutine pool
[   0.861]    INFO ziti/controller/network.(*Network).showOptions: network = {
  "CreateCircuitRetries": 2,
  "CycleSeconds": 60,
  "EnableLegacyLinkMgmt": false,
  "InitialLinkLatency": 65000000000,
  "IntervalAgeThreshold": 0,
  "MetricsReportInterval": 60000000000,
  "MinRouterCost": 10,
  "PendingLinkTimeout": 10000000000,
  "RouteTimeout": 10000000000,
  "RouterConnectChurnLimit": 60000000000,
  "RouterComm": {
    "QueueSize": 100,
    "MaxWorkers": 100
  },
  "Smart": {
    "RerouteFraction": 0.02,
    "RerouteCap": 4,
    "MinCostDelta": 15
  }
}
[   0.861]    INFO ziti/controller/webapis.NewFabricManagementApiFactory: initializing management api factory with 0 xmgmt instances
[   0.861]    INFO ziti/controller.(*Controller).showOptions: ctrl = {
  "OutQueueSize": 4,
  "MaxQueuedConnects": 1,
  "MaxOutstandingConnects": 16,
  "ConnectTimeout": 5000000000,
  "DelayRxStart": false,
  "WriteTimeout": 0,
  "MessageStrategy": null,
  "NewListener": null,
  "AdvertiseAddress": {},
  "RouterHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  },
  "PeerHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  }
}
[   0.861]    INFO ziti/controller/server.NewController: edge controller instance id: cm6kyt2u20000u3o56qf1rnve
[   0.861]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   0.862]    INFO ziti/controller/sync_strats.(*InstantStrategy).Initialize: {logSize=[10000] listenerBufferSizes=[1000]} initialized controller router data model
[   0.864]    INFO ziti/controller/sync_strats.(*InstantStrategy).BuildAll.func1: {index=[0]} initialized router data model from db
[   0.864]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
[   0.865]    INFO ziti/controller/server.(*Controller).Run: starting edge
[   0.865]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] maxQueueSize=[1] maxWorkers=[16] idleTime=[10s] poolType=[pool.listener.ctrl]} starting goroutine pool
[   0.865]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[   0.868]    INFO ziti/controller/server.(*Controller).checkEdgeInitialized: edge initialized
[   1.059]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:1280 for server client-management with APIs: [edge-management edge-client fabric edge-oidc]
[   1.060]    INFO ziti/controller/network.(*Network).Run: started
[   4.231]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:56358]} handshake failed
[   5.080]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:56450]} handshake failed
[   5.090]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:56454]} handshake failed
[   5.296]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[172.19.0.1:56506] error=[remote error: tls: bad certificate]} handshake failed
[   5.334]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[172.19.0.1:56522] error=[remote error: tls: bad certificate]} handshake failed
[   5.334]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:56510]} handshake failed
[   7.961]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48536] error=[remote error: tls: bad certificate]} handshake failed
[   7.973]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48544] error=[remote error: tls: bad certificate]} handshake failed
[   8.047]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48548] error=[remote error: tls: bad certificate]} handshake failed
[   8.055]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48564] error=[remote error: tls: bad certificate]} handshake failed
[   8.186]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48574]} handshake failed
[   8.199]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48588] error=[remote error: tls: bad certificate]} handshake failed
[   8.289]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48594] error=[remote error: tls: bad certificate]} handshake failed
[   8.346]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48604] error=[remote error: tls: bad certificate]} handshake failed
[   8.488]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48618]} handshake failed
[   8.502]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48622] error=[remote error: tls: bad certificate]} handshake failed
[   8.712]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48638] error=[remote error: tls: bad certificate]} handshake failed
[   8.793]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48646]} handshake failed
[   8.833]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48662] error=[remote error: tls: bad certificate]} handshake failed
[   8.889]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48678]} handshake failed
[   9.024]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48690] error=[remote error: tls: bad certificate]} handshake failed
[   9.100]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48704]} handshake failed
[   9.173]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48712] error=[remote error: tls: bad certificate]} handshake failed
[   9.251]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {error=[remote error: tls: bad certificate] remote=[172.19.0.1:48720]} handshake failed
[   9.255]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48736] error=[remote error: tls: bad certificate]} handshake failed
[   9.334]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48748] error=[remote error: tls: bad certificate]} handshake failed
[   9.344]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48754] error=[remote error: tls: bad certificate]} handshake failed
[   9.420]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48758] error=[remote error: tls: bad certificate]} handshake failed
[   9.586]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48770] error=[remote error: tls: bad certificate]} handshake failed
[   9.659]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48774] error=[remote error: tls: bad certificate]} handshake failed
[   9.676]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:6262]: {remote=[172.19.0.1:48788] error=[remote error: tls: bad certificate]} handshake failed
[   9.690]   ERROR transport/v2/tls.(*shared

So the handhake failed errors, I would suspect, are because a tunneler/identity/router of some kind was turned on in the past and it is now trying to reconnect to the new url. You'll have to find that identity/tunneler/router and turn it off or you'll see this over and over. The certificate being presented is not valid because the quickstart will regenerate the full PKI... That make sense? Other than that, it appears the controller is online properly.

As for the ziti console waiting for the key. I'm not sure what happened there. Can you share the exact set of steps you performed to get to this state? Maybe I can spot a problem -- or mabye there's some issue we need to sort out.

thx for your support, I am using the quickstart - docker-compose - approach

step1. remove the conatiners , volumes, check export env,

user@overlay:~/openziti$ ps -elf |grep ziti
0 S user     1563780 1557122  0  80   0 -  1636 pipe_r 08:20 pts/0    00:00:00 grep --color=auto ziti
user@overlay:~/openziti$ docker-compose down -v
[+] Running 12/12
 ✔ Container openziti-ziti-edge-router-1                Removed                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-console-1                    Removed                                                                                                                                                 10.3s
 ✔ Container openziti-ziti-controller-init-container-1  Removed                                                                                                                                                  0.0s
 ✔ Container openziti-web-test-blue-1                   Removed                                                                                                                                                  0.2s
 ✔ Container openziti-ziti-fabric-router-br-1           Removed                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-private-blue-1               Removed                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-private-red-1                Removed                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-edge-router-wss-1            Removed                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-controller-1                 Removed                                                                                                                                                 10.4s
 ✔ Volume openziti_ziti-fs                              Removed                                                                                                                                                  0.0s
 ✔ Network openziti_zitired                             Removed                                                                                                                                                  0.2s
 ✔ Network openziti_zitiblue                            Removed                                                                                                                                                  0.4s
user@overlay:~/openziti$

root@overlay:/home/user/openziti# docker volume ls
DRIVER    VOLUME NAME

user@overlay:~/openziti$ export
declare -x PATH="/home/user/.ziti/quickstart/overlay/ziti-bin/ziti-v1.1.15/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
declare -x PWD="/home/user/openziti"
declare -x XDG_DATA_DIRS="/usr/local/share:/usr/share:/var/lib/snapd/desktop"

declare -x ZITI_CTRL_ADVERTISED_ADDRESS="ziti-ctrl.ddns.net"
declare -x ZITI_CTRL_EDGE_ADVERTISED_ADDRESS="zitiedge-ctrl.ddns.net"
declare -x ZITI_CTRL_NAME="ziti-controller"
declare -x ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION="10080"
declare -x ZITI_IMAGE="openziti/quickstart"
declare -x ZITI_INTERFACE="0.0.0.0"
declare -x ZITI_PWD="xxxxxxxx"
declare -x ZITI_ROUTER_ENROLLMENT_DURATION="10080"
declare -x ZITI_USER="admin"
declare -x ZITI_VERSION="latest"
user@overlay:~/openziti$



user@overlay:~/openziti$ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
user@overlay:~/openziti$

user@overlay:~/openziti$ docker-compose up -d
[+] Running 12/12
 ✔ Network openziti_zitiblue                            Created                                                                                                                                                  0.1s
 ✔ Network openziti_zitired                             Created                                                                                                                                                  0.1s
 ✔ Volume "openziti_ziti-fs"                            Created                                                                                                                                                  0.0s
 ✔ Container openziti-web-test-blue-1                   Started                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-controller-1                 Healthy                                                                                                                                                  0.0s
 ✔ Container openziti-ziti-console-1                    Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-fabric-router-br-1           Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-private-red-1                Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-controller-init-container-1  Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-private-blue-1               Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-edge-router-1                Started                                                                                                                                                  0.1s
 ✔ Container openziti-ziti-edge-router-wss-1            Started                                                                                                                                                  0.1s
user@overlay:~/openziti$

user@overlay:~/openziti$ docker ps -a
CONTAINER ID   IMAGE                        COMMAND                   CREATED          STATUS                      PORTS                                              NAMES
b4e184ed0d77   openziti/quickstart:latest   "/bin/bash /var/open…"    59 seconds ago   Up 34 seconds                                                                  openziti-ziti-private-blue-1
252e6a650a3a   openziti/quickstart:latest   "/bin/bash /var/open…"    59 seconds ago   Up 34 seconds                                                                  openziti-ziti-private-red-1
be9e03c421dd   openziti/quickstart:latest   "/var/openziti/scrip…"    59 seconds ago   Exited (0) 32 seconds ago                                                      openziti-ziti-controller-init-container-1
d49cf0774b80   openziti/quickstart:latest   "/bin/bash /var/open…"    59 seconds ago   Up 34 seconds               0.0.0.0:3022->3022/tcp, 0.0.0.0:10080->10080/tcp   openziti-ziti-edge-router-1
e74d1b46de79   openziti/quickstart:latest   "/bin/bash /var/open…"    59 seconds ago   Up 34 seconds               0.0.0.0:3023->3023/tcp, 0.0.0.0:10081->10081/tcp   openziti-ziti-edge-router-wss-1
148c903a1a8c   openziti/quickstart:latest   "/bin/bash /var/open…"    59 seconds ago   Up 34 seconds                                                                  openziti-ziti-fabric-router-br-1
b7e68d5575be   openziti/zac                 "/usr/src/app/run-za…"    59 seconds ago   Up 34 seconds               1408/tcp, 0.0.0.0:8443->8443/tcp                   openziti-ziti-console-1
897cbe2599e4   openziti/quickstart:latest   "/var/openziti/scrip…"    59 seconds ago   Up 58 seconds (healthy)     0.0.0.0:1280->1280/tcp, 0.0.0.0:6262->6262/tcp     openziti-ziti-controller-1
b2011abf5e71   openziti/hello-world         "/bin/sh -c 'echo \"h…"   59 seconds ago   Up 58 seconds (healthy)     0.0.0.0:80->8000/tcp                               openziti-web-test-blue-1
user@overlay:~/openziti$ docker logs openziti-ziti-console-1
waiting for server key to exist...
waiting for server key to exist...

But I can only see that these certificates are re-generated

ZITI_ENV_FILE overridden: /persistent/ziti.env
Your OpenZiti environment has been set up successfully.

A file with all pertinent environment values was created here: /persistent/ziti.env

Generating PKI
Creating CA: ziti-controller-root-ca
Success

Creating CA: ziti-edge-controller-root-ca
Success

Creating CA: ziti-signing-root-ca
Success

Creating intermediate: ziti-controller-root-ca ziti-controller-intermediate 1
Using CA name:  ziti-controller-root-ca
Success

Creating intermediate: ziti-edge-controller-root-ca ziti-edge-controller-intermediate 1
Using CA name:  ziti-edge-controller-root-ca
Success

Creating intermediate: ziti-signing-root-ca ziti-signing-intermediate_grandparent_intermediate 2
Using CA name:  ziti-signing-root-ca
Success

Creating intermediate: ziti-signing-intermediate_grandparent_intermediate ziti-signing-intermediate 1
Using CA name:  ziti-signing-intermediate_grandparent_intermediate
Success


Creating server cert from ca: ziti-controller-intermediate for localhost,ziti,ziti-controller / 127.0.0.1
Using CA name:  ziti-controller-intermediate
Success
Creating client cert from ca: ziti-controller-intermediate for localhost,ziti,ziti-controller
Using CA name:  ziti-controller-intermediate
Success

Creating server cert from ca: ziti-edge-controller-intermediate for localhost,ziti,zitiedge-ctrl.ddns.net / 127.0.0.1
Using CA name:  ziti-edge-controller-intermediate
Success
Creating client cert from ca: ziti-edge-controller-intermediate for localhost,ziti,zitiedge-ctrl.ddns.net
Using CA name:  ziti-edge-controller-intermediate
Success

PKI generated successfully



Thanks @Reinhard. When you did the up, did you copy the commands as shown on the page? I just did all the steps again, but I added a pull in there to make sure I had the latest/greatest and things were fine for me. Maybe this is a OS-related issue? (I use linux on WSL):

mkdir /tmp/docker
cd /tmp/docker
curl -so docker-compose.yaml https://get.openziti.io/dock/docker-compose.yml
curl -so .env https://get.openziti.io/dock/.env
docker compose pull
docker compose --project-name docker up 

Also, the reason I asked if you copied and ran those commands from the page is because I noticed in your down you're not adding the project name, not that it should matter imo, but who knows if somehow that is your difference or maybe it was the pull?

Can you ensure you have the latest and pull and let me know what OS you're running this on?

user@overlay:~/openziti$ docker images
REPOSITORY                 TAG       IMAGE ID       CREATED         SIZE
openziti/quickstart        latest    ed5779e048e0   8 days ago      336MB
openziti/zac               latest    2afd2c32654b   11 days ago     1.01GB
openziti/ziti-router       latest    d4cf5c47864c   4 months ago    557MB
openziti/ziti-controller   latest    cb1ebfdb6135   4 months ago    619MB
busybox                    latest    af4709625109   4 months ago    4.27MB
openziti/hello-world       latest    63e90784502f   17 months ago   4.26MB
user@overlay:~/openziti$ cat .env
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest

# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
ZITI_USER=admin
ZITI_PWD=Ro1b00s1!

ZITI_INTERFACE=0.0.0.0

# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=zitiedge-ctrl.ddns.net
ZITI_CTRL_ADVERTISED_ADDRESS=ziti-ctrl.ddns.net
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
#ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
#ZITI_CTRL_ADVERTISED_PORT=8440

# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080

# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
#ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-edge-router
#ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public
user@overlay:~/openziti$

Ah, I lost sight of the fact that you mention it's only when you change the advertised address.

change the ziti-console environment like this and it should start up:

      - ZAC_SERVER_CERT_CHAIN=/persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-intermediate.cert
      - ZAC_SERVER_KEY=/persistent/pki/ziti-edge-controller-intermediate/keys/ziti-edge-controller-intermediate.key

I think the problem is that the console container has this environments but

"Env": [
                "ZAC_SERVER_KEY=/persistent/pki/zitiedge-ctrl.ddns.net-intermediate/keys/zitiedge-ctrl.ddns.net-server.key",

                "ZITI_CTRL_ADVERTISED_ADDRESS=ziti-ctrl.ddns.net",

                "ZAC_SERVER_CERT_CHAIN=/persistent/pki/zitiedge-ctrl.ddns.net-intermediate/certs/zitiedge-ctrl.ddns.net-server.cert",

but this pki console container path does not exist, the directory /persistent/pki/zitiedge-ctrl.ddns.net-intermediate has not been created or?

user@overlay:~/openziti/backup$ docker exec -it ziti-ziti-console-1 bash
root@b68fb4d6be6f:/usr/src/app# cd /persistent/pki/
root@b68fb4d6be6f:/persistent/pki# ls -al
total 52
drwxr-xr-x 9 2171 2171 4096 Feb  3 14:56 .
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:57 ..
-rw-r--r-- 1 2171 2171 4155 Feb  3 14:56 cas.pem
-rw-r--r-- 1 2171 2171 4208 Feb  3 14:56 signing.pem
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-ctrl.ddns.net-intermediate
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-ctrl.ddns.net-root-ca
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-edge-controller-intermediate
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-edge-controller-root-ca
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-signing-intermediate
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-signing-intermediate_grandparent_intermediate
drwxr-xr-x 5 2171 2171 4096 Feb  3 14:56 ziti-signing-root-ca
root@b68fb4d6be6f:/persistent/pki#

I'm confused. Did you modify the compose file ZAC_SERVER_CERT_CHAIN and ZAC_SERVER_KEY and were you successful, and you're just providing more information? Or are you still stuck? I also tested this and things appeared to work for me so I'm just not sure what the situation is now.

Yes, the there appears to be a bug with the quickstart, complicated docker compose example when running with a different advertised address. You should be able to work around that by addressing the compose file keys.

What is the current issue you're facing?

sorry to confuse, I provided more Information because I think this can be a bug. But your workaround

 - ZAC_SERVER_CERT_CHAIN=/persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-intermediate.cert
  - ZAC_SERVER_KEY=/persistent/pki/ziti-edge-controller-intermediate/keys/ziti-edge-controller-intermediate.key

solves the issue, so that I can connect to the console.

1 Like