[SELF-HOSTED] Fresh setup `i/o timeout`

Hi, I couldn't find more information elsewhere.

I just did a fresh installation on a docker based self-hosted setup.

I can't make any shares working. The command is stuck on i/o timeout.

I did try:

  • disable firewall
  • use -insecure
  • tested various shares (public, private, reserved)
 $ zrok share public 8000

[   0.792]    INFO sdk-golang/ziti.(*listenerManager).createSessionWithBackoff: {session token=[0dd77050-883c-4790-a63f-d6dc74548404]} new service session
[  15.794]   ERROR sdk-golang/ziti.(*ContextImpl).connectEdgeRouter: {router=[quickstart-router]} dial tcp <REDACTED>:3022: i/o timeout

Server side:

caddy-1                  | 2025/03/19 14:07:34.960	INFO	http.log.access.log0	handled request	{"request": {"remote_ip": "<REDACTED>", "remote_port": "64823", "client_ip": "<REDACTED>", "proto": "HTTP/2.0", "method": "GET", "host": "zrok.tunnel.<REDACTED>", "uri": "/api/v1/overview", "headers": {"Accept-Encoding": ["gzip, deflate, br, zstd"], "Cookie": ["REDACTED"], "X-Token": ["q0TA4oTRnsTe"], "Sec-Fetch-Site": ["same-origin"], "Referer": ["https://zrok.tunnel.<REDACTED>/"], "Priority": ["u=1, i"], "Sec-Fetch-Mode": ["cors"], "Accept-Language": ["fr-FR,fr;q=0.9,en;q=0.8"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Accept": ["*/*"], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua": ["\"Chromium\";v=\"134\", \"Not:A-Brand\";v=\"24\", \"Google Chrome\";v=\"134\""], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["empty"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "zrok.tunnel.<REDACTED>"}}, "bytes_read": 0, "user_id": "", "duration": 0.090032476, "size": 555, "status": 200, "resp_headers": {"Date": ["Wed, 19 Mar 2025 14:07:34 GMT"], "Content-Length": ["555"], "Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Type": ["application/zrok.v1+json"]}}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:821","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).runRefreshes","level":"debug","msg":"refreshing services","time":"2025-03-19T14:07:40.399Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:668","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).refreshServices","level":"debug","msg":"checking if service updates available","time":"2025-03-19T14:07:40.411Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:686","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).refreshServices","level":"debug","msg":"refreshing services","time":"2025-03-19T14:07:40.587Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:492","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).processServiceUpdates","level":"debug","msg":"processing service updates with 1 services","time":"2025-03-19T14:07:40.610Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/edge/types.go:165","func":"github.com/openziti/sdk-golang/ziti/edge.ParseServiceConfig","level":"debug","msg":"no service config of type intercept.v1 defined for service","serviceId":"68tWe0GLfcw1ns7StkmnPa","serviceName":"d0xhrjqxzwdi","time":"2025-03-19T14:07:40.613Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/edge/types.go:165","func":"github.com/openziti/sdk-golang/ziti/edge.ParseServiceConfig","level":"debug","msg":"no service config of type ziti-tunneler-client.v1 defined for service","serviceId":"68tWe0GLfcw1ns7StkmnPa","serviceName":"d0xhrjqxzwdi","time":"2025-03-19T14:07:40.615Z"}
zrok-frontend-1          | {"file":"/__w/zrok/zrok/endpoints/publicProxy/http.go:194","func":"github.com/openziti/zrok/endpoints/publicProxy.NewHTTP.shareHandler.func2","level":"debug","msg":"auth scheme none 'd0xhrjqxzwdi'","time":"2025-03-19T14:08:17.180Z"}
zrok-frontend-1          | {"file":"/__w/zrok/zrok/endpoints/publicProxy/http.go:127","func":"github.com/openziti/zrok/endpoints/publicProxy.newServiceProxy.hostTargetReverseProxy.func4","level":"debug","msg":"auth model: map[auth_scheme:none basic_auth:\u003cnil\u003e interstitial:true oauth:\u003cnil\u003e]","time":"2025-03-19T14:08:17.193Z"}
zrok-frontend-1          | {"file":"/__w/zrok/zrok/endpoints/publicProxy/http.go:132","func":"github.com/openziti/zrok/endpoints/publicProxy.newServiceProxy.hostTargetReverseProxy.func4","level":"info","msg":"[d0xhrjqxzwdi] -\u003e /","time":"2025-03-19T14:08:17.200Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:1097","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).DialWithOptions","level":"debug","msg":"connecting with session","sessionId":"cm8fzvwgl007j01p9ni8wte82","sessionToken":"80bd45d5-df0b-4dc6-b667-0dc4b6ca3580","time":"2025-03-19T14:08:17.226Z"}
zrok-frontend-1          | {"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/ziti.go:1315","func":"github.com/openziti/sdk-golang/ziti.(*ContextImpl).getEdgeRouterConn","level":"debug","msg":"selected router[quickstart-router@tls:ziti.tunnel.<REDACTED>:3022] for best latency(15 ms)","sessionId":"cm8fzvwgl007j01p9ni8wte82","time":"2025-03-19T14:08:17.231Z"}
ziti-quickstart-1        | [ 412.544]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators] routerId=[zx1W1usSQ] operation=[create.circuit]} responded with error
ziti-quickstart-1        | [ 412.549] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {chSeq=[22] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators] token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] edgeSeq=[0] connId=[6] type=[EdgeConnectType]} failed to dial fabric
zrok-frontend-1          | {"connId":6,"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/edge/network/conn.go:582","func":"github.com/openziti/sdk-golang/ziti/edge/network.(*edgeConn).close","level":"debug","marker":"8sqnKfPV","msg":"close: begin","time":"2025-03-19T14:08:17.290Z"}
zrok-frontend-1          | {"connId":6,"file":"/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v0.23.44/ziti/edge/network/conn.go:603","func":"github.com/openziti/sdk-golang/ziti/edge/network.(*edgeConn).close","level":"debug","marker":"8sqnKfPV","msg":"close: end","time":"2025-03-19T14:08:17.294Z"}
zrok-frontend-1          | {"file":"/__w/zrok/zrok/

Except that, I can create user, access the web console, show the diagrams.

Thank you.

This means the device where you ran zrok share public is unable to reach the Ziti router port, e.g., ziti.${ZROK_DNS_ZONE}:${ZITI_ROUTER_PORT}, since you're following the zrok Docker self-hosting guide.

On the Docker host, ensure you're publishing ZITI_ROUTER_PORT. That's one of the "secure ports" that must be published on an IP address that's reachable by all parts of your system, typically a public IP.

Also, on the Docker host, ensure ZITI_ROUTER_PORT is allowed INPUT on the firewall so incoming packets can reach Docker's published port.


Correction: another possibility is that the networking is good but the router port is closed because the router isn't listening for some reason. With docker compose ps ziti-quickstart you can verify that container is running continually, not crashing.

Hi, thank you for your answer. In deed, i checked the firewall, and the port, I think they are fine.

Here are the logs of ziti:

DEBUG: run command is: ziti edge quickstart --home /home/ziggy/quickstart  --ctrl-address ziti.tunnel.<REDACTED> --ctrl-port 80 --router-address ziti.tunnel.<REDACTED> --router-port 3022 --password <REDACTED>
[   0.002]    INFO ziti/ziti/cmd/edge.(*QuickstartOpts).run: permanent --home '/home/ziggy/quickstart' will not be removed on exit
Starting controller...
Controller running...
[   0.046]    INFO ziti/ziti/controller.run: {version=[v1.4.3] arch=[amd64] build-date=[2025-03-04T16:52:50Z] go-version=[go1.23.5] os=[linux] revision=[de60092629f9] nodeId=[quickstart]} starting ziti-controller
[   0.078]    INFO ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1: edge datastore is up to date at version 37
[   1.264]    INFO storage/boltz.(*DbImpl).GetTimelineId.func1: checking timeline id. reset required? false timelineId: XbygXwhHR
[   1.308]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[100] minWorkers=[0] maxWorkers=[100] idleTime=[30s] poolType=[pool.router.messaging]} starting goroutine pool
[   1.309]    INFO ziti/controller/network.(*Network).showOptions: network = {
  "CreateCircuitRetries": 2,
  "CycleSeconds": 60,
  "EnableLegacyLinkMgmt": false,
  "InitialLinkLatency": 65000000000,
  "IntervalAgeThreshold": 0,
  "MetricsReportInterval": 60000000000,
  "MinRouterCost": 10,
  "PendingLinkTimeout": 10000000000,
  "RouteTimeout": 10000000000,
  "RouterConnectChurnLimit": 60000000000,
  "RouterComm": {
    "QueueSize": 100,
    "MaxWorkers": 100
  },
  "Smart": {
    "RerouteFraction": 0.02,
    "RerouteCap": 4,
    "MinCostDelta": 15
  }
}
[   1.311]    INFO ziti/controller/webapis.NewFabricManagementApiFactory: initializing management api factory with 0 xmgmt instances
[   1.311]    INFO ziti/controller.(*Controller).showOptions: ctrl = {
  "OutQueueSize": 4,
  "MaxQueuedConnects": 1,
  "MaxOutstandingConnects": 16,
  "ConnectTimeout": 5000000000,
  "DelayRxStart": false,
  "WriteTimeout": 0,
  "MessageStrategy": null,
  "NewListener": null,
  "AdvertiseAddress": {},
  "RouterHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  },
  "PeerHeartbeatOptions": {
    "sendInterval": 10000000000,
    "checkInterval": 1000000000,
    "closeUnresponsiveTimeout": 30000000000
  }
}
[   1.312]    INFO ziti/controller/server.NewController: edge controller instance id: cm8fzqj9d000001p9w10vkhq5
[   1.313]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   1.327]    INFO ziti/controller/sync_strats.(*InstantStrategy).Initialize: {logSize=[10000] listenerBufferSizes=[1000]} initialized controller router data model
[   1.356]    INFO ziti/controller/sync_strats.(*InstantStrategy).BuildAll.func1: {index=[44]} initialized router data model from db
[   1.357]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
[   1.378]    INFO ziti/controller/server.(*Controller).Run: starting edge
[   1.380]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] maxWorkers=[16] maxQueueSize=[1] idleTime=[10s] poolType=[pool.listener.ctrl]} starting goroutine pool
[   1.405]    INFO ziti/controller/server.(*Controller).checkEdgeInitialized: edge initialized
[   1.405]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[   1.612]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:80]: {remote=[172.18.0.3:47832] error=[tls: client didn't provide a certificate]} handshake failed
[   1.826]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:80]: {remote=[172.18.0.3:47842] error=[tls: client didn't provide a certificate]} handshake failed
[   1.901]    INFO ziti/controller/network.(*Network).Run: started
[   1.902]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:80 for server client-management with APIs: [edge-management edge-client fabric edge-oidc]
[   1.938]    INFO ziti/ziti/cmd/edge.(*QuickstartOpts).run: Controller online. Continuing...
[   1.974]    INFO ziti/router.(*EdgeConfig).loadCsr: loaded csr info from configuration file at path [edge.csr]
[   1.974] WARNING ziti/router.(*EdgeConfig).LoadEdgeConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[   1.974]    INFO ziti/ziti/router.run: {version=[v1.4.3] go-version=[go1.23.5] routerId=[zx1W1usSQ] arch=[amd64] configFile=[/home/ziggy/quickstart/quickstart-router.yaml] revision=[de60092629f9] build-date=[2025-03-04T16:52:50Z] os=[linux]} starting ziti router
[   1.975]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[0] maxQueueSize=[1000] idleTime=[30s] maxWorkers=[32] poolType=[pool.link.dialer]} starting goroutine pool
[   1.975]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] maxQueueSize=[1000] poolType=[pool.rdm.handler] maxWorkers=[1] minWorkers=[1]} starting goroutine pool
[   1.977]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[128] idleTime=[30s] maxQueueSize=[1000] poolType=[pool.route.handler] minWorkers=[0]} starting goroutine pool
[   1.977]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] maxQueueSize=[1] maxWorkers=[50] poolType=[pool.terminator_validation] minWorkers=[0]} starting goroutine pool
[   1.983]    INFO ziti/router/forwarder.(*Faulter).run: started
[   1.983]    INFO ziti/router/forwarder.(*Scanner).run: started
[   1.985]    INFO ziti/router/state.(*ManagerImpl).LoadRouterModel: {path=[/home/ziggy/quickstart/quickstart-router.yaml.proto.gzip] index=[44]} loaded router model from file
[   1.987]    INFO ziti/router/state.(*ManagerImpl).SetRouterDataModel: {index=[44]} replacing router data model
[   1.987]    INFO ziti/router/state.(*ManagerImpl).SetRouterDataModel: {index=[44]} router data model replacement complete, old: 0x0, new: 0xc002570540
[   1.989]   ERROR ziti/ziti/router.run: {error=[gops: agent already listening at: /tmp/gops-agent.1.sock]} unable to start CLI agent
[   1.990]    INFO ziti/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":5000000000,"DelayRxStart":false,"WriteTimeout":0,"MessageStrategy":null}
[   1.990]    INFO ziti/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"IntervalAgeThreshold":0,"MessageQueueSize":10}
[   1.991]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.rate_limiter] minWorkers=[0] maxWorkers=[15] idleTime=[30s] maxQueueSize=[5000]} starting goroutine pool
[   1.991]    INFO ziti/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[   1.991]    INFO ziti/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[   1.991] WARNING ziti/router/xlink_transport.loadListenerConfig: {error=[no network interface found for 0.0.0.0] addr=[tls:0.0.0.0:3022]} unable to get interface for address
[   1.993]    INFO ziti/router/xlink_transport.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.listener.link] idleTime=[10s] maxQueueSize=[1] minWorkers=[1] maxWorkers=[16]} starting goroutine pool
[   1.994]    INFO ziti/router.(*Router).startXlinkListeners: started Xlink listener with binding [transport] advertising [tls:ziti.tunnel.<REDACTED>:3022]
[   1.996]    INFO ziti/router/xgress_edge.(*Factory).CreateListener: xgress edge listener options: mtu=0
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=4194304
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=28
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
lookupApiSessionTimeout=5s
lookupSessionTimeout=5s
channel.outQueueSize=4
channel.connectTimeout=5s
channel.maxOutstandingConnects=16
channel.maxQueuedConnects=1

[   1.997]    INFO ziti/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:3022]} starting channel listener
[   1.997]    INFO ziti/router/xgress_edge.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.listener.xgress_edge] minWorkers=[1] maxQueueSize=[1] maxWorkers=[16] idleTime=[10s]} starting goroutine pool
[   1.998]    INFO ziti/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:3022]
[   1.998]    INFO ziti/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[   1.998]    INFO ziti/router.(*Router).getInitialCtrlEndpoints: loading controller endpoints from [/home/ziggy/quickstart/endpoints]
[   1.999]    INFO ziti/router.(*Router).startControlPlane: router configured with 1 controller endpoints
[   2.000]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[0]} sync all subscribers
[   2.002]    INFO ziti/router/xgress_edge.(*Acceptor).Run: starting
[   2.002]    INFO ziti/router/env.(*networkControllers).UpdateControllerEndpoints: {endpoint=[map[tls:ziti.tunnel.<REDACTED>:80:{}]]} adding new ctrl endpoint
[   2.002]    INFO ziti/router/env.(*networkControllers).connectToControllerWithBackoff: {endpoint=[tls:ziti.tunnel.<REDACTED>:80]} starting connection attempts
Router is available on ziti.tunnel.<REDACTED>:3022
[   2.073]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[EOF] remote=[172.18.0.3:42078]} handshake failed

=======================================================================================
controller and router started.
    controller located at  : ziti.tunnel.<REDACTED>:80
    router located at      : ziti.tunnel.<REDACTED>:3022
    config dir located at  : /home/ziggy/quickstart
    configured trust domain: quickstart
    instance pid           : 1
=======================================================================================
[   2.091]    INFO ziti/controller/handler_ctrl.(*CtrlAccepter).Bind: {revision=[de60092629f9] arch=[amd64] buildDate=[2025-03-04T16:52:50Z] os=[linux] routerId=[zx1W1usSQ] version=[v1.4.3]} accepted new router connection
[   2.093]    INFO ziti/controller/env.(*Broker).RouterConnected: {routerId=[zx1W1usSQ] routerName=[quickstart-router] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34]} broker detected edge router with id zx1W1usSQ connecting
[   2.093]    INFO ziti/controller/sync_strats.(*InstantStrategy).RouterConnected: {sync_strategy=[instant] routerId=[zx1W1usSQ] routerName=[quickstart-router] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34] syncStatus=[SYNC_QUEUED]} edge router connected, adding to sync routerConnectedQueue
[   2.094]    INFO ziti/controller/network.(*routerChangedEvent).handle: {routerId=[zx1W1usSQ] connected=[true]} calculating router updates for router
[   2.095]    INFO ziti/controller/sync_strats.(*InstantStrategy).hello: {routerChannelIsOpen=[true] routerName=[quickstart-router] routerTxId=[ZjkwL5D03] strategy=[instant] routerId=[zx1W1usSQ] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34]} edge router sync starting
[   2.095]    INFO ziti/controller/sync_strats.(*InstantStrategy).hello: {routerName=[quickstart-router] routerTxId=[ZjkwL5D03] strategy=[instant] routerId=[zx1W1usSQ] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34] syncStatus=[SYNC_HELLO] routerChannelIsOpen=[true]} sending edge router hello
[   2.097]    INFO ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func3: {endpoint=[tls:ziti.tunnel.<REDACTED>:80]} successfully connected to controller
[   2.098]    INFO ziti/router/link.(*linkRegistryImpl).NotifyOfReconnect: {ctrlId=[quickstart]} resending link states after reconnect
[   2.098]    INFO ziti/router/xgress_edge_tunnel.NewFactoryWrapper.func2: router data model enabled, using xgress_edge_tunnel_v2
[   2.098]    INFO ziti/router/xgress_edge_tunnel_v2.(*tunneler).NotifyIdentityEvent: identity updated zx1W1usSQ, eventType: identity.full-state
[   2.098]    INFO ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription: {ctrlId=[quickstart] prevCtrlId=[]} no current data model subscription active, subscribing
[   2.099]    INFO ziti/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1: received server hello, replying
[   2.100]    INFO ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates: {ctrlId=[quickstart] currentIndex=[44] renew=[false]} subscribed to new controller for router data model changes
[   2.101]    INFO ziti/router/xgress_edge.(*Factory).NotifyOfReconnect: control channel reconnected, re-establishing hosted services
[   2.102]    INFO ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive: {routerId=[zx1W1usSQ] index=[44] timelineId=[XbygXwhHR]} data model subscription request received
[   2.103]    INFO ziti/controller/sync_strats.(*InstantStrategy).ReceiveClientHello: {data=[map[]] routerId=[zx1W1usSQ] protocols=[[tls]] routerVersion=[v1.4.3] buildDate=[2025-03-04T16:52:50Z] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34] dataModelIndex=[44] arch=[amd64] version=[v1.4.3] os=[linux] routerChannelIsOpen=[true] listeners=[[address:{value:"tls:0.0.0.0:3022" protocol:"tls" hostname:"0.0.0.0" port:3022} advertise:{value:"ziti.tunnel.<REDACTED>:3022" protocol:"tls" hostname:"ziti.tunnel.<REDACTED>" port:3022}]] strategy=[instant] routerTxId=[ZjkwL5D03] routerName=[quickstart-router] revision=[de60092629f9] serverVersion=[v1.4.3] protocolPorts=[[3022]] routerIndex=[44]} edge router sent hello
[   2.103]    INFO ziti/controller/sync_strats.(*InstantStrategy).synchronize: {routerChannelIsOpen=[true] strategy=[instant] routerName=[quickstart-router] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34] routerTxId=[ZjkwL5D03] routerId=[zx1W1usSQ] SupportsRouterModel=[true]} started synchronizing edge router
[   2.119]    INFO ziti/controller/sync_strats.(*RouterSender).handleSyncRequest: {renew=[false] routerId=[zx1W1usSQ] currentIndex=[44] routerName=[quickstart-router] requestedIndex=[44] subscriptionDuration=[2025-03-19 14:06:26.853913591 +0000 UTC m=+302.303753185] timelineId=[XbygXwhHR]} data model subscription started
[   2.125]    INFO ziti/controller/sync_strats.(*InstantStrategy).synchronize.func1: {strategy=[instant] SupportsRouterModel=[true] routerId=[zx1W1usSQ] routerName=[quickstart-router] routerFingerprint=[2dd4fc0ce900ef81bbe8c8111b35d1a5f2116f34] routerChannelIsOpen=[true] routerTxId=[ZjkwL5D03]} exiting synchronization, final status: SYNC_DONE
[   2.125]   ERROR ziti/common.(*RouterDataModel).ApplyChangeSet: {index=[0] synthetic=[false] entries=[1] action=[Create] type=[*edge_ctrl_pb.DataState_Event_PublicKey] error=[out of order event detected, currentIndex: 44, receivedIndex: 0, type :*common.ForgetfulEventCache]} could not apply change set
[   2.127]   ERROR ziti/common.(*RouterDataModel).ApplyChangeSet: {type=[*edge_ctrl_pb.DataState_Event_PublicKey] error=[out of order event detected, currentIndex: 44, receivedIndex: 0, type :*common.ForgetfulEventCache] entries=[1] action=[Create] index=[0] synthetic=[false]} could not apply change set
[   2.127]    INFO ziti/router/state.(*apiSessionAddedHandler).instantSync: {strategy=[instant]} first api session syncId [cm8fzqjw7000101p9sro4r1g5], starting
[   2.127]    INFO ziti/router/state.(*apiSessionSyncTracker).Add: received api session sync chunk 0, isLast=true
[   2.175]    INFO ziti/router/state.(*ManagerImpl).StartHeartbeat: heartbeat starting
[   2.175]    INFO ziti/router/xgress_edge.(*CertExpirationChecker).Run: waiting 8589h30m9.089503586s to renew certificates
[   3.128]    INFO ziti/router/state.(*apiSessionAddedHandler).applySync: finished synchronizing api sessions [count: 23, syncId: cm8fzqjw7000101p9sro4r1g5, duration: 135.607µs]
[   7.868]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:80]: {error=[remote error: tls: bad certificate] remote=[172.18.0.5:55210]} handshake failed
[  49.655]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[  49.681]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[  49.711]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[  49.728]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[  75.818]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {operation=[create.circuit] token=[734980be-223e-48c0-93f9-bedebec533e8] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators] routerId=[zx1W1usSQ]} responded with error
[  75.819] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {edgeSeq=[0] token=[734980be-223e-48c0-93f9-bedebec533e8] connId=[1] type=[EdgeConnectType] chSeq=[1] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators]} failed to dial fabric
[  75.963]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {routerId=[zx1W1usSQ] token=[734980be-223e-48c0-93f9-bedebec533e8] operation=[create.circuit] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators]} responded with error
[  75.966] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {type=[EdgeConnectType] token=[734980be-223e-48c0-93f9-bedebec533e8] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators] chSeq=[3] edgeSeq=[0] connId=[2]} failed to dial fabric
[  78.569]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {routerId=[zx1W1usSQ] operation=[create.circuit] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators] token=[734980be-223e-48c0-93f9-bedebec533e8]} responded with error
[  78.573] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {chSeq=[5] edgeSeq=[0] token=[734980be-223e-48c0-93f9-bedebec533e8] error=[service 2S9rlnJmSuajOmuyxrVWDA has no terminators] connId=[3] type=[EdgeConnectType]} failed to dial fabric
[ 206.731]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 206.774]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 206.804]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 206.831]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 213.706]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 213.724]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 213.754]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 213.781]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 251.708]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {routerId=[zx1W1usSQ] operation=[create.circuit] token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators]} responded with error
[ 251.710] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] connId=[4] type=[EdgeConnectType] chSeq=[12] edgeSeq=[0] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators]} failed to dial fabric
[ 251.853]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] routerId=[zx1W1usSQ] operation=[create.circuit] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators]} responded with error
[ 251.855] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {edgeSeq=[0] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators] token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] connId=[5] type=[EdgeConnectType] chSeq=[14]} failed to dial fabric
[ 302.105]    INFO ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription: {ctrlId=[quickstart] prevCtrlId=[quickstart]} current data model subscription expired, resubscribing
[ 302.108]    INFO ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates: {currentIndex=[56] renew=[true] ctrlId=[quickstart]} subscribed to new controller for router data model changes
[ 302.117]    INFO ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive: {index=[56] timelineId=[XbygXwhHR] routerId=[zx1W1usSQ]} data model subscription request received
[ 302.118]    INFO ziti/controller/sync_strats.(*RouterSender).handleSyncRequest: {currentIndex=[56] renew=[true] routerId=[zx1W1usSQ] routerName=[quickstart-router] subscriptionDuration=[2025-03-19 14:11:26.854110446 +0000 UTC m=+602.303950080] timelineId=[XbygXwhHR] requestedIndex=[56]} data model subscription started
[ 412.544]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{zx1W1usSQ}->u{classic}->i{zx1W1usSQ/WJ9K}]: {token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators] routerId=[zx1W1usSQ] operation=[create.circuit]} responded with error
[ 412.549] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{EX1QgusSW/WMbR}]: {chSeq=[22] error=[service 68tWe0GLfcw1ns7StkmnPa has no terminators] token=[80bd45d5-df0b-4dc6-b667-0dc4b6ca3580] edgeSeq=[0] connId=[6] type=[EdgeConnectType]} failed to dial fabric
[ 511.413]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 511.424]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 511.460]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 511.486]    INFO ziti/common.syncAllSubscribersEvent.process: {subs=[1]} sync all subscribers
[ 514.070]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:80]: {remote=[<REDACTED>:64853] error=[EOF]} handshake failed
[ 632.102]    INFO ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription: {ctrlId=[quickstart] prevCtrlId=[quickstart]} current data model subscription expired, resubscribing
[ 632.103]    INFO ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates: {currentIndex=[60] renew=[true] ctrlId=[quickstart]} subscribed to new controller for router data model changes
[ 632.107]    INFO ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive: {index=[60] timelineId=[XbygXwhHR] routerId=[zx1W1usSQ]} data model subscription request received
[ 632.109]    INFO ziti/controller/sync_strats.(*RouterSender).handleSyncRequest: {requestedIndex=[60] currentIndex=[60] renew=[true] subscriptionDuration=[2025-03-19 14:16:56.843500578 +0000 UTC m=+932.293340172] timelineId=[XbygXwhHR] routerName=[quickstart-router] routerId=[zx1W1usSQ]} data model subscription started
[ 811.748]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:80]: {remote=[<REDACTED>:64926] error=[EOF]} handshake failed
[ 962.101]    INFO ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription: {ctrlId=[quickstart] prevCtrlId=[quickstart]} current data model subscription expired, resubscribing
[ 962.101]    INFO ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates: {ctrlId=[quickstart] currentIndex=[60] renew=[true]} subscribed to new controller for router data model changes
[ 962.107]    INFO ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive: {routerId=[zx1W1usSQ] timelineId=[XbygXwhHR] index=[60]} data model subscription request received
[ 962.109]    INFO ziti/controller/sync_strats.(*RouterSender).handleSyncRequest: {subscriptionDuration=[2025-03-19 14:22:26.843894783 +0000 UTC m=+1262.293734377] requestedIndex=[60] currentIndex=[60] renew=[true] timelineId=[XbygXwhHR] routerId=[zx1W1usSQ] routerName=[quickstart-router]} data model subscription started
[1292.102]    INFO ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription: {ctrlId=[quickstart] prevCtrlId=[quickstart]} current data model subscription expired, resubscribing
[1292.105]    INFO ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates: {ctrlId=[quickstart] currentIndex=[60] renew=[true]} subscribed to new controller for router data model changes
[1292.110]    INFO ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive: {routerId=[zx1W1usSQ] index=[60] timelineId=[XbygXwhHR]} data model subscription request received
[1292.111]    INFO ziti/controller/sync_strats.(*RouterSender).handleSyncRequest: {timelineId=[XbygXwhHR] currentIndex=[60] renew=[true] routerId=[zx1W1usSQ] routerName=[quickstart-router] requestedIndex=[60] subscriptionDuration=[2025-03-19 14:27:56.846723104 +0000 UTC m=+1592.296562738]} data model subscription started
[1420.125]    INFO ziti/ziti/controller.waitForShutdown: shutting down ziti-controller
[1420.131]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: shutting down...
[1420.131]    INFO ziti/controller/server.(*Controller).Shutdown: edge controller: stopped
[1420.131]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: shutting down...
[1420.133]    INFO ziti/controller/server.(*Controller).Shutdown: fabric controller: stopped
[1420.133]    INFO ziti/controller/server.(*Controller).Shutdown: shutdown complete
Signal to shutdown received
Environment left intact at: /home/ziggy/quickstart

Ok, I found the issue. It was my security group. 3022 port wasn't allowed. My bad. Now it is working :wink:

1 Like