Failed to connect to ER (on a separate box from the controller)

1

Hello,

I'm trying to set up the following:

ziti-diagram
ziti-diagram865×385 15.4 KB

I have the controller up and running with ZAC and I'm able to log in (both with ZAC and ziti CLI from the router as well as with the ziti-edge-tunneler binary on my Linux machine)

The controller has no edge routers yet when I run the following commands from the edge router machine:

cd $ZITI_HOME
ziti edge login https://srv-dmz-ziti-controller:1280 -u admin
ziti edge create edge-router $(hostname) -o $(hostname).jwt -t -a 'public'
ziti create config router edge --routerName $(hostname) > $(hostname).yaml
ziti router enroll $(hostname).yaml --jwt $(hostname).jwt

The config in question is:

v: 3

identity:
  cert:             "/var/lib/ziti/srv-dmz-ziti-edge-router.cert"
  server_cert:      "/var/lib/ziti/srv-dmz-ziti-edge-router.server.chain.cert"
  key:              "/var/lib/ziti/srv-dmz-ziti-edge-router.key"
  ca:               "/var/lib/ziti/srv-dmz-ziti-edge-router.cas"
  #alt_server_certs:
  #  - server_cert:  ""
  #    server_key:   ""

ctrl:
  endpoint:             tls:srv-dmz-ziti-controller:6262

link:
  dialers:
    - binding: transport
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:our.public.dns:10080
      options:
        outQueueSize:   4

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: our.public.dns:3022
      connectTimeoutMs: 5000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host #tproxy|host



edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - localhost
        - our.public.dns
        - srv-dmz-ziti-edge-router
      ip:
        - "127.0.0.1"
        - "::1"
        - "A.B.C.D"


#transport:
#  ws:
#    writeTimeout: 10
#    readTimeout: 5
#    idleTimeout: 120
#    pongTimeout: 60
#    pingInterval: 54
#    handshakeTimeout: 10
#    readBufferSize: 4096
#    writeBufferSize: 4096
#    enableCompression: true

forwarder:
  latencyProbeInterval: 0
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32

I'm then able to run the router with systemd and it shows as online and connected in ZAC:

image
image1516×129 14.4 KB

The logs are showing:

Aug 20 12:39:24 srv-dmz-ziti-edge-router systemd[1]: Started ziti-edge-router.service - Ziti-Router for srv-dmz-ziti-edge-router.
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"arch":"amd64","build-date":"2024-07-16T13:15:20Z","configFile":"/var/lib/ziti/srv-dmz-ziti-edge-router.yaml","file":"github.com/openziti/ziti/ziti/router/run.go:71","func":"github.com/openziti/ziti/ziti/router.run","go-version":"go1.22.5","level":"info","msg":"starting ziti router","os":"linux","revision":"94013fe4af89","routerId":"WTrYehgRHT","time":"2024-08-20T12:39:25.007Z","version":"v1.1.7"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":32,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.link.dialer","time":"2024-08-20T12:39:25.008Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/forwarder/faulter.go:78","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"info","msg":"started","time":"2024-08-20T12:39:25.008Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":128,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.route.handler","time":"2024-08-20T12:39:25.008Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1,"maxWorkers":50,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.terminator_validation","time":"2024-08-20T12:39:25.009Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:154","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"info","msg":"cached data model file set to: /var/lib/ziti/srv-dmz-ziti-edge-router.yaml.json.gzip","time":"2024-08-20T12:39:25.009Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:171","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"warning","msg":"Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]","time":"2024-08-20T12:39:25.009Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/forwarder/scanner.go:52","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).run","level":"info","msg":"started","time":"2024-08-20T12:39:25.008Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:346","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"ctrl = {\"OutQueueSize\":4,\"MaxQueuedConnects\":1,\"MaxOutstandingConnects\":16,\"ConnectTimeout\":5000000000,\"DelayRxStart\":false,\"WriteTimeout\":0}","time":"2024-08-20T12:39:25.009Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:352","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"metrics = {\"ReportInterval\":60000000000,\"IntervalAgeThreshold\":0,\"MessageQueueSize\":10}","time":"2024-08-20T12:39:25.010Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":5000,"maxWorkers":15,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.rate_limiter","time":"2024-08-20T12:39:25.010Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:660","func":"github.com/openziti/ziti/router.(*Router).initializeHealthChecks","level":"info","msg":"starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s","time":"2024-08-20T12:39:25.010Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:481","func":"github.com/openziti/ziti/router.(*Router).startXlinkDialers","level":"info","msg":"started Xlink dialer with binding [transport]","time":"2024-08-20T12:39:25.010Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"addr":"tls:0.0.0.0:10080","error":"no network interface found for 0.0.0.0","file":"github.com/openziti/ziti/router/xlink_transport/config.go:76","func":"github.com/openziti/ziti/router/xlink_transport.loadListenerConfig","level":"warning","msg":"unable to get interface for address","time":"2024-08-20T12:39:25.011Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/router/xlink_transport.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1","idleTime":10000000000,"level":"info","maxQueueSize":1,"maxWorkers":16,"minWorkers":1,"msg":"starting goroutine pool","poolType":"pool.listener.link","time":"2024-08-20T12:39:25.011Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:506","func":"github.com/openziti/ziti/router.(*Router).startXlinkListeners","level":"info","msg":"started Xlink listener with binding [transport] advertising [tls:our.public.dns:10080]","time":"2024-08-20T12:39:25.011Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"address":{},"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:87","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen","level":"info","msg":"starting channel listener","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1","idleTime":10000000000,"level":"info","maxQueueSize":1,"maxWorkers":16,"minWorkers":1,"msg":"starting goroutine pool","poolType":"pool.listener.xgress_edge","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [edge] at [tls:0.0.0.0:3022]","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/xgress_edge/accept.go:126","func":"github.com/openziti/ziti/router/xgress_edge.(*Acceptor).Run","level":"info","msg":"starting","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [tunnel] at []","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:722","func":"github.com/openziti/ziti/router.(*Router).getInitialCtrlEndpoints","level":"info","msg":"controller endpoints file [/var/lib/ziti/endpoints] doesn't exist. Using initial endpoints from config","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/router.go:555","func":"github.com/openziti/ziti/router.(*Router).startControlPlane","level":"info","msg":"router configured with 1 controller endpoints","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"endpoint":{"tls:srv-dmz-ziti-controller:6262":{}},"file":"github.com/openziti/ziti/router/env/ctrls.go:95","func":"github.com/openziti/ziti/router/env.(*networkControllers).UpdateControllerEndpoints","level":"info","msg":"adding new ctrl endpoint","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"endpoint":"tls:srv-dmz-ziti-controller:6262","file":"github.com/openziti/ziti/router/env/ctrls.go:134","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff","level":"info","msg":"starting connection attempts","time":"2024-08-20T12:39:25.012Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"endpoint":"tls:srv-dmz-ziti-controller:6262","file":"github.com/openziti/ziti/router/env/ctrls.go:140","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func3","level":"info","msg":"successfully connected to controller","time":"2024-08-20T12:39:25.062Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"ctrlId":"srv-dmz-ziti-controller","file":"github.com/openziti/ziti/router/link/link_registry.go:306","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).NotifyOfReconnect","level":"info","msg":"resending link states after reconnect","time":"2024-08-20T12:39:25.062Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/xgress_edge/factory.go:77","func":"github.com/openziti/ziti/router/xgress_edge.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-08-20T12:39:25.062Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/factory.go:56","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-08-20T12:39:25.062Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/handler_edge_ctrl/hello.go:82","func":"github.com/openziti/ziti/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1","level":"info","msg":"received server hello, replying","time":"2024-08-20T12:39:25.064Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:203","func":"github.com/openziti/ziti/router/state.(*apiSessionAddedHandler).instantSync","level":"info","msg":"first api session syncId [cm02am01400v5m1v0ui343zbp], starting","strategy":"instant","time":"2024-08-20T12:39:25.066Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:268","func":"github.com/openziti/ziti/router/state.(*apiSessionSyncTracker).Add","level":"info","msg":"received api session sync chunk 0, isLast=true","time":"2024-08-20T12:39:25.066Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/state/manager.go:604","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).StartHeartbeat","level":"info","msg":"heartbeat starting","time":"2024-08-20T12:39:25.091Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/tunneler.go:71","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*tunneler).Start","level":"info","mode":"host","msg":"creating interceptor","time":"2024-08-20T12:39:25.091Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/xgress_edge/certchecker.go:124","func":"github.com/openziti/ziti/router/xgress_edge.(*CertExpirationChecker).Run","level":"info","msg":"waiting 8591h59m36.907691764s to renew certificates","time":"2024-08-20T12:39:25.092Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:59","func":"github.com/openziti/ziti/tunnel/dns.flushDnsCaches","level":"info","msg":"dns caches flushed","time":"2024-08-20T12:39:25.097Z"}
Aug 20 12:39:25 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:89","func":"github.com/openziti/ziti/tunnel/dns.NewDnsServer","level":"info","msg":"starting dns server...","time":"2024-08-20T12:39:25.097Z"}
Aug 20 12:39:26 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:124","func":"github.com/openziti/ziti/router/state.(*apiSessionAddedHandler).applySync","level":"info","msg":"finished synchronizing api sessions [count: 4, syncId: cm02am01400v5m1v0ui343zbp, duration: 33.144µs]","time":"2024-08-20T12:39:26.066Z"}
Aug 20 12:39:27 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:119","func":"github.com/openziti/ziti/tunnel/dns.NewDnsServer","level":"info","msg":"dns server running at 127.0.0.1:53","time":"2024-08-20T12:39:27.097Z"}
Aug 20 12:39:27 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:271","func":"github.com/openziti/ziti/tunnel/dns.(*resolver).AddHostname","level":"info","msg":"adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver","time":"2024-08-20T12:39:27.097Z"}
Aug 20 12:39:27 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:130","func":"github.com/openziti/ziti/tunnel/dns.NewDnsServer","level":"error","msg":"system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test: no such host\n\nziti-tunnel runs an internal DNS server which must be first in the host's\nresolver configuration. On systems that use NetManager/dhclient, this can\nbe achieved by adding the following to /etc/dhcp/dhclient.conf:\n\n    prepend domain-name-servers 127.0.0.1:53;\n\n","time":"2024-08-20T12:39:27.101Z"}
Aug 20 12:39:27 srv-dmz-ziti-edge-router ziti[5606]: {"file":"github.com/openziti/ziti/tunnel/intercept/iputils.go:51","func":"github.com/openziti/ziti/tunnel/intercept.SetDnsInterceptIpRange","level":"info","msg":"dns intercept IP range: 100.64.0.1 - 100.127.255.255","time":"2024-08-20T12:39:27.101Z"}

However, when I create an identity for myself and add it to my ziti-edge-tunnel instance, it doesn't want to connect to the edge router:

{
  "Success": true,
  "Data": {
    "Active": true,
    "Duration": 1000889,
    "StartTime": "2024-08-20T10:41:56.921759Z",
    "Identities": [
      {
        "Name": "rbgtk",
        "Identifier": "/opt/ziti/etc/identities/me.json",
        "FingerPrint": "org",
        "Active": true,
        "Loaded": true,
        "Config": {
          "ztAPI": "https://our.public.dns:1280"
        },
        "ControllerVersion": "v1.1.7",
        "IdFileStatus": true,
        "MfaEnabled": false,
        "MfaNeeded": false,
        "Metrics": {
          "Up": 0,
          "Down": 0
        },
        "MfaMinTimeout": 0,
        "MfaMaxTimeout": 0,
        "MfaMinTimeoutRem": 0,
        "MfaMaxTimeoutRem": 0,
        "MinTimeoutRemInSvcEvent": 0,
        "MaxTimeoutRemInSvcEvent": 0,
        "Deleted": false,
        "Notified": false
      },
      {
        ...
      }
    ],
    "IpInfo": {
      "Ip": "100.64.0.1",
      "Subnet": "255.192.0.0",
      "MTU": 65535,
      "DNS": "100.64.0.2"
    },
    "LogLevel": "info",
    "ServiceVersion": {
      "Version": "v1.1.1",
      "BuildDate": "Tue-08/13/2024-13:08:01-UTC"
    },
    "TunIpv4": "100.64.0.1",
    "TunIpv4Mask": 10,
    "AddDns": false,
    "ApiPageSize": 25
  },
  "Code": 0
}

The logs for my ziti-edge-tunneler say the following:

Aug 20 12:41:56 archlinux systemd[1]: Started Ziti Edge Tunnel.
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: About to run tunnel service... ziti-edge-tunnel
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-sdk:utils.c:201 ziti_log_set_level() set log level: root=3/INFO
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 1.0.7 @g9fa5162(HEAD) starting at (2024-08-20T10:41:56.921)
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-edge-tunnel:instance-config.c:86 load_tunnel_status_from_file() Loading config file from /var/lib/ziti/config.json
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-sdk:utils.c:201 ziti_log_set_level() set log level: root=3/INFO
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.1.1)
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO tunnel-cbs:ziti_dns.c:164 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1724 run_tunneler_loop() Loading identity files from /opt/ziti/etc/identities
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1125 load_identities() loading identity file: me.json
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1650 make_socket_path() effective group set to 'ziti' (gid=962)
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.012]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.012]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/opt/ziti/etc/identities/me.json]
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.012]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/opt/ziti/etc/identities/me.json]
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.012]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1140 load_id_cb() identity[/opt/ziti/etc/identities/me.json] loaded
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.018]    INFO ziti-sdk:ziti.c:457 ziti_init_async() ztx[0] using tlsuv[v0.29.5], tls[Mbed TLS 2.28.5]
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.018]    INFO ziti-sdk:ziti.c:459 ziti_init_async() ztx[1] Loading ziti context with controller[https://our.public.dns:1280]
Aug 20 12:41:56 archlinux ziti-edge-tunnel[22148]: (22148)[        0.018]    INFO ziti-sdk:ziti.c:936 ziti_re_auth_with_cb() ztx[1] starting to re-auth with ctrl[https://our.public.dns:1280] api_session_status[0] api_session_expired[TRUE]
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.130]    INFO ziti-sdk:ziti.c:1697 version_cb() ztx[1] connected to controller https://our.public.dns:1280 version v1.1.7(94013fe4af89 2024-07-16T13:15:20Z)
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.142]    INFO ziti-sdk:ziti.c:1588 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.142]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[rbgtk] connected to controller
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.239]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.239]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[0] reconnecting NOW
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.348]    INFO ziti-sdk:ziti.c:1588 ziti_set_api_session() ztx[1] api session set, setting api_session_timer to 1740s
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.348]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[rbgtk] connected to controller
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.348]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1170 on_event() ztx[/opt/ziti/etc/identities/me.json] context event : status is OK
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.452]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[1] (srv-dmz-ziti-edge-router) new channel for ztx[1] identity[rbgtk]
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.452]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[rbgtk] added edge router srv-dmz-ziti-edge-router@our.public.dns
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.452]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[1] reconnecting NOW
Aug 20 12:41:57 archlinux ziti-edge-tunnel[22148]: (22148)[        0.493]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.5|aec0d3b9acfb|2024-07-02T18:20:02Z|linux|amd64
Aug 20 12:41:58 archlinux ziti-edge-tunnel[22148]: (22148)[        1.143]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
Aug 20 12:41:58 archlinux ziti-edge-tunnel[22148]: (22148)[        1.348]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[1] first run or potential controller restart detected
Aug 20 12:42:17 archlinux ziti-edge-tunnel[22148]: (22148)[       20.493]   ERROR ziti-sdk:channel.c:709 ch_connect_timeout() ch[1] connect timeout
Aug 20 12:42:17 archlinux ziti-edge-tunnel[22148]: (22148)[       20.493]    INFO ziti-sdk:channel.c:775 reconnect_channel() ch[1] reconnecting in 7501ms (attempt = 1)
Aug 20 12:42:17 archlinux ziti-edge-tunnel[22148]: (22148)[       20.493]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[srv-dmz-ziti-edge-router] [-125/operation canceled]

Edited "our.public.dns" for privacy and removed log entries from a working install for another organization

I know I could just revert back to a single machine with both controller and edge router on the same box, but my team's asked me to see if there's a way to separate the two. From all I gather, I reckon it should be possible, so I must be doing something wrong.

Thanks in advance!

2

The edge router is advertising srv-dmz-ziti-edge-router for edge clients to connect to it. I expect your ziti-edge-tunnel cannot route to that hostname. edit the router's config file and locate the edge binding section and update it with a routable address and restart the router and your tunneler will connect.

example:

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:8442
    options:
      advertise: ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8442
3

Hi,

Thanks for the feedback!

I've confirmed that our.public.dns is advertised in the router's yaml config, and restarted the service. Unfortunately, to no avail.

Then I installed everything from scratch, following the documentation for "Hosted anywhere" and even with a single expressInstall, it's now advertising srv-dmz-ziti-controller-edge-router even though the configs are as follows:

Controller

v: 3

#trace:
#  path: "our.public.dns.trace"

#profile:
#  memory:
#    path: ctrl.memprof



db:                     "/var/lib/ziti/db/ctrl.db"
# uncomment and configure to enable HA
# raft:
#   dataDir:         "/var/lib/ziti/raft"
#   minClusterSize:  1


identity:
  cert:        "/var/lib/ziti/pki/srv-dmz-ziti-controller-intermediate/certs/srv-dmz-ziti-controller-client.cert"
  server_cert: "/var/lib/ziti/pki/srv-dmz-ziti-controller-intermediate/certs/srv-dmz-ziti-controller-server.chain.pem"
  key:         "/var/lib/ziti/pki/srv-dmz-ziti-controller-intermediate/keys/srv-dmz-ziti-controller-server.key"
  ca:          "/var/lib/ziti/pki/cas.pem"
  #alt_server_certs:
  #  - server_cert:  ""
  #    server_key:   ""

trustDomain: quickstart

# Network Configuration
#
# Configure how the controller will establish and manage the overlay network, and routing operations on top of
# the network.
#
#network:

  # routeTimeoutSeconds controls the number of seconds the controller will wait for a route attempt to succeed.
  #routeTimeoutSeconds:  10

  # createCircuitRetries controls the number of retries that will be attempted to create a path (and terminate it)
  # for new circuits.
  #createCircuitRetries: 2

  # pendingLinkTimeoutSeconds controls how long we'll wait before creating a new link between routers where
  # there isn't an established link, but a link request has been sent
  #pendingLinkTimeoutSeconds: 10

  # Defines the period that the controller re-evaluates the performance of all of the circuits
  # running on the network.
  #
  #cycleSeconds:         15

  # Sets router minimum cost. Defaults to 10
  #minRouterCost: 10

  # Sets how often a new control channel connection can take over for a router with an existing control channel connection
  # Defaults to 1 minute
  #routerConnectChurnLimit: 1m

  # Sets the latency of link when it's first created. Will be overwritten as soon as latency from the link is actually
  # reported from the routers. Defaults to 65 seconds.
  #initialLinkLatency: 65s

  #smart:
    #
    # Defines the fractional upper limit of underperforming circuits that are candidates to be re-routed. If
    # smart routing detects 100 circuits that are underperforming, and `smart.rerouteFraction` is set to `0.02`,
    # then the upper limit of circuits that will be re-routed in this `cycleSeconds` period will be limited to
    # 2 (2% of 100).
    #
    #rerouteFraction:    0.02
    #
    # Defines the hard upper limit of underperforming circuits that are candidates to be re-routed. If smart
    # routing detects 100 circuits that are underperforming, and `smart.rerouteCap` is set to `1`, and
    # `smart.rerouteFraction` is set to `0.02`, then the upper limit of circuits that will be re-routed in this
    # `cycleSeconds` period will be limited to 1.
    #
    #rerouteCap:         4

# the endpoint that routers will connect to the controller over.
ctrl:
  options:
    advertiseAddress: tls:srv-dmz-ziti-controller:6262
  # (optional) settings
  # set the maximum number of connect requests that are buffered and waiting to be acknowledged (1 to 5000, default 1)
  #maxQueuedConnects:      1
  # the maximum number of connects that have  begun hello synchronization (1 to 1000, default 16)
  #maxOutstandingConnects: 16
  # the number of milliseconds to wait before a hello synchronization fails and closes the connection (30ms to 60000ms, default: 5000ms)
  #connectTimeoutMs:       5000
  listener:             tls:0.0.0.0:6262

#metrics:
#  influxdb:
#    url:                http://localhost:8086
#    database:           ziti

# xctrl_example
#
#example:
#  enabled:              false
#  delay:                5s

healthChecks:
  boltCheck:
    # How often to try entering a bolt read tx. Defaults to 30 seconds
    interval: 30s
    # When to time out the check. Defaults to 20 seconds
    timeout: 20s
    # How long to wait before starting the check. Defaults to 30 seconds
    initialDelay: 30s

# By having an 'edge' section defined, the ziti-controller will attempt to parse the edge configuration. Removing this
# section, commenting out, or altering the name of the section will cause the edge to not run.
edge:
  # This section represents the configuration of the Edge API that is served over HTTPS
  api:
    #(optional, default 90s) Alters how frequently heartbeat and last activity values are persisted
    # activityUpdateInterval: 90s
    #(optional, default 250) The number of API Sessions updated for last activity per transaction
    # activityUpdateBatchSize: 250
    # sessionTimeout - optional, default 30m
    # The number of minutes before an Edge API session will time out. Timeouts are reset by
    # API requests and connections that are maintained to Edge Routers
    sessionTimeout: 30m
    # address - required
    # The default address (host:port) to use for enrollment for the Client API. This value must match one of the addresses
    # defined in this Controller.WebListener.'s bindPoints.
    address: our.public.dns:1280
  # This section is used to define option that are used during enrollment of Edge Routers, Ziti Edge Identities.
  enrollment:
    # signingCert - required
    # A Ziti Identity configuration section that specifically makes use of the cert and key fields to define
    # a signing certificate from the PKI that the Ziti environment is using to sign certificates. The signingCert.cert
    # will be added to the /.well-known CA store that is used to bootstrap trust with the Ziti Controller.
    signingCert:
      cert: /var/lib/ziti/pki/signing.pem
      key:  /var/lib/ziti/pki/srv-dmz-ziti-controller-signing-intermediate/keys/srv-dmz-ziti-controller-signing-intermediate.key
    # edgeIdentity - optional
    # A section for identity enrollment specific settings
    edgeIdentity:
      # duration - optional, default 180m
      # The length of time that a Ziti Edge Identity enrollment should remain valid. After
      # this duration, the enrollment will expire and no longer be usable.
      duration: 180m
    # edgeRouter - Optional
    # A section for edge router enrollment specific settings.
    edgeRouter:
      # duration - optional, default 180m
      # The length of time that a Ziti Edge Router enrollment should remain valid. After
      # this duration, the enrollment will expire and no longer be usable.
      duration: 180m

# web
# Defines webListeners that will be hosted by the controller. Each webListener can host many APIs and be bound to many
# bind points.
web:
  # name - required
  # Provides a name for this listener, used for logging output. Not required to be unique, but is highly suggested.
  - name: client-management
    # bindPoints - required
    # One or more bind points are required. A bind point specifies an interface (interface:port string) that defines
    # where on the host machine the webListener will listen and the address (host:port) that should be used to
    # publicly address the webListener(i.e. mydomain.com, localhost, 127.0.0.1). This public address may be used for
    # incoming address resolution as well as used in responses in the API.
    bindPoints:
      #interface - required
      # A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces
      - interface: 0.0.0.0:1280
        # address - required
        # The public address that external incoming requests will be able to resolve. Used in request processing and
        # response content that requires full host:port/path addresses.
        address: our.public.dns:1280
    # identity - optional
    # Allows the webListener to have a specific identity instead of defaulting to the root 'identity' section.
    identity:
      ca:          "/var/lib/ziti/pki/srv-dmz-ziti-controller-edge-controller-root-ca/certs/srv-dmz-ziti-controller-edge-controller-root-ca.cert"
      key:         "/var/lib/ziti/pki/srv-dmz-ziti-controller-edge-controller-intermediate/keys/our.public.dns-server.key"
      server_cert: "/var/lib/ziti/pki/srv-dmz-ziti-controller-edge-controller-intermediate/certs/our.public.dns-server.chain.pem"
      cert:        "/var/lib/ziti/pki/srv-dmz-ziti-controller-edge-controller-intermediate/certs/our.public.dns-client.cert"
      #alt_server_certs:
      #- server_cert: ""
      #  server_key:  ""

    # options - optional
    # Allows the specification of webListener level options - mainly dealing with HTTP/TLS settings. These options are
    # used for all http servers started by the current webListener.
    options:
      # idleTimeoutMs - optional, default 5000ms
      # The maximum amount of idle time in milliseconds allowed for pipelined HTTP requests. Setting this too high
      # can cause resources on the host to be consumed as clients remain connected and idle. Lowering this value
      # will cause clients to reconnect on subsequent HTTPs requests.
      idleTimeout: 5000ms  #http timeouts, new
      # readTimeoutMs - optional, default 5000ms
      # The maximum amount of time in milliseconds http servers will wait to read the first incoming requests. A higher
      # value risks consuming resources on the host with clients that are acting bad faith or suffering from high latency
      # or packet loss. A lower value can risk losing connections to high latency/packet loss clients.
      readTimeout: 5000ms
      # writeTimeoutMs - optional, default 100000ms
      # The total maximum time in milliseconds that the http server will wait for a single requests to be received and
      # responded too. A higher value can allow long-running requests to consume resources on the host. A lower value
      # can risk ending requests before the server has a chance to respond.
      writeTimeout: 100000ms
      # minTLSVersion - optional, default TLS1.2
      # The minimum version of TSL to support
      minTLSVersion: TLS1.2
      # maxTLSVersion - optional, default TLS1.3
      # The maximum version of TSL to support
      maxTLSVersion: TLS1.3
    # apis - required
    # Allows one or more APIs to be bound to this webListener
    apis:
      # binding - required
      # Specifies an API to bind to this webListener. Built-in APIs are
      #   - edge-management
      #   - edge-client
      #   - fabric-management
      - binding: edge-management
        # options - arg optional/required
        # This section is used to define values that are specified by the API they are associated with.
        # These settings are per API. The example below is for the 'edge-api' and contains both optional values and
        # required values.
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
      - binding: zac
        options:
          location: ./ziti-console
          indexFile: index.html

Router

v: 3

identity:
  cert:             "/var/lib/ziti/srv-dmz-ziti-controller-edge-router.cert"
  server_cert:      "/var/lib/ziti/srv-dmz-ziti-controller-edge-router.server.chain.cert"
  key:              "/var/lib/ziti/srv-dmz-ziti-controller-edge-router.key"
  ca:               "/var/lib/ziti/srv-dmz-ziti-controller-edge-router.cas"
  #alt_server_certs:
  #  - server_cert:  ""
  #    server_key:   ""

ctrl:
  endpoint:             tls:srv-dmz-ziti-controller:6262

link:
  dialers:
    - binding: transport
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:our.public.dns:10080
      options:
        outQueueSize:   4

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: our.public.dns:3022
      connectTimeoutMs: 5000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host #tproxy|host



edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - localhost
        - our.public.dns
        - srv-dmz-ziti-controller
      ip:
        - "127.0.0.1"
        - "::1"
        - "194.78.10.122"


#transport:
#  ws:
#    writeTimeout: 10
#    readTimeout: 5
#    idleTimeout: 120
#    pongTimeout: 60
#    pingInterval: 54
#    handshakeTimeout: 10
#    readBufferSize: 4096
#    writeBufferSize: 4096
#    enableCompression: true

forwarder:
  latencyProbeInterval: 0
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32

Ziti Edge Tunnel

Aug 20 16:56:38 archlinux ziti-edge-tunnel[36227]: (36227)[      527.662]   ERROR ziti-sdk:channel.c:709 ch_connect_timeout() ch[0] connect timeout
Aug 20 16:56:38 archlinux ziti-edge-tunnel[36227]: (36227)[      527.662]    INFO ziti-sdk:channel.c:775 reconnect_channel() ch[0] reconnecting in 123383ms (attempt = 8)
Aug 20 16:56:38 archlinux ziti-edge-tunnel[36227]: (36227)[      527.662]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[0] failed to connect to ER[srv-dmz-ziti-controller-edge-router] [-125/operation canceled]
4

Can you DM me your actual IP or DNS and I'll probe it from my side?

Also looking at the controller config, it appears that the variables weren't all set, or maybe you edited the config? endpoint: tls:srv-dmz-ziti-controller:6262 should be an actual FQDN of some sort. Is that a redacted version or is that actual?

I am pretty sure one or more of the expected IP/ports are just not available. It's hard to konw which if i can't probe them. :frowning:

5

Hi, sure, I will DM to you!

6

after chatting on DM, the router ports are not responding to openssl commands.

Verify the router ports are responding with:

openssl s_client -connect your.fqdn.here:3022
openssl s_client -connect your.fqdn.here:10080

(obviously replace the your.fqdn.here and ports) and I think it'll be fine! :slight_smile: