Not able to install router via helm chart to connect to ziti controller at port 443 but able to connect via router script

Can you help with the following:
router installation

helm upgrade --install "private-router2" openziti/ziti-router \
--namespace ziti \
--set-file enrollmentJwt=./router2.jwt \
--set edge.advertisedHost=private-router2-edge.ziti.svc.cluster.local \
--set linkListeners.transport.service.enabled=false \
--set tunnel.mode=host \
--set ctrl.endpoint="https://ztnctrl.domain:443"

We currently installed ziti controller with the following configuration

clientApi:
  advertisedHost: ztnclient.domain.co
  advertisedPort: 443
  service:
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"

ctrlPlane:
  advertisedHost: ztnctrl.domain.co
  advertisedPort: 443
  service:
    enabled: true
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"

    ```

Observation:

We are able to see the terminators been created , we can ping the intercept domain but we cant establish the connection 

Here are the router logs 

{"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{pVAaE}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"6y8qFhsHrA3xZh8nuQxICr","time":"2024-07-16T18:03:02.267Z"}
{"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{pVAaE}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"4RNsgVriNjbZhMQqOvWWtu","time":"2024-07-16T18:03:02.267Z"}
{"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{pVAaE}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"7jR72S6WcGcp72yRE4ydpb","time":"2024-07-16T18:03:02.267Z"}


here are the controller logs 

{"_channels":["selectPath"],"apiSessionId":"clyoo76xbeuzm0d9t3eqk3ayc","attemptNumber":1,"circuitId":"TnR3BIjPv","error":"error creating route for [s/TnR3BIjPv] on [r/el0zwLEXz] (error creating route for [c/TnR3BIjPv]: dial tcp: lookup hello.hello-toy.svc on 127.0.0.53:53: server misbehaving)","file":"github.com/openziti/ziti/controller/network/network.go:597","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"2H19pztKRO7rJvADRTKCBU","serviceName":"router2-service","sessionId":"clyope1xqewjm0d9t4elfhg8r","time":"2024-07-16T18:03:12.809Z"}
{"_channels":["establishPath"],"apiSessionId":"clyoo76xbeuzm0d9t3eqk3ayc","attemptNumber":2,"circuitId":"TnR3BIjPv","file":"github.com/openziti/ziti/controller/network/routesender.go:196","func":"github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/el0zwLEXz] for attempt [#1] of [s/TnR3BIjPv] (error creating route for [c/TnR3BIjPv]: dial tcp: lookup hello.hello-toy.svc on 127.0.0.53:53: server misbehaving)","serviceId":"2H19pztKRO7rJvADRTKCBU","sessionId":"clyope1xqewjm0d9t4elfhg8r","time":"2024-07-16T18:03:12.815Z"}
{"_channels":["selectPath"],"apiSessionId":"clyoo76xbeuzm0d9t3eqk3ayc","attemptNumber":2,"circuitId":"TnR3BIjPv","error":"error creating route for [s/TnR3BIjPv] on [r/el0zwLEXz] (error creating route for [c/TnR3BIjPv]: dial tcp: lookup hello.hello-toy.svc on 127.0.0.53:53: server misbehaving)","file":"github.com/openziti/ziti/controller/network/network.go:597","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"2H19pztKRO7rJvADRTKCBU","serviceName":"router2-service","sessionId":"clyope1xqewjm0d9t4elfhg8r","time":"2024-07-16T18:03:12.816Z"}
{"_channels":["selectPath"],"apiSessionId":"clyoo76xbeuzm0d9t3eqk3ayc","attemptNumber":3,"circuitId":"TnR3BIjPv","file":"github.com/openziti/ziti/controller/network/network.go:606","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"circuit creation failed after [2] attempts, sending cleanup unroutes","serviceId":"2H19pztKRO7rJvADRTKCBU","serviceName":"router2-service","sessionId":"clyope1xqewjm0d9t4elfhg8r","time":"2024-07-16T18:03:12.816Z"}
{"_context":"ch{el0zwLEXz}-\u003eu{classic}-\u003ei{dmJdM}","error":"exceeded maximum [2] retries creating circuit [c/TnR3BIjPv]: error creating route for [s/TnR3BIjPv] on [r/el0zwLEXz] (error creating route for [c/TnR3BIjPv]: dial tcp: lookup hello.hello-toy.svc on 127.0.0.53:53: server misbehaving)","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:79","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"el0zwLEXz","time":"2024-07-16T18:03:12.816Z","token":"70b30a91-553c-46a7-bd0a-24252b4ff0bb"}

Welcome back, @bazooka720. Your router chart value ctrl.endpoint must match your controller chart value ctrlPlane.advertisedHost.

It seems like two different issues appear in your report:

  1. Unable to install router chart because router can't connect to controller - this may be addressed by aligning those chart values I mentioned
  2. A client of a Ziti service can't "establish the connection" - this sounds like the service isn't fully configured yet. It's not possible to evaluate a Ziti service with ping. Ziti services are TCP or UDP, not ICMP, so any echo request may be answered by an intercepting tun device, but does not represent an echo response from the other end of the service. You must test with a TCP or UDP-enabled tool other than ping. One way to evaluate service health is ziti edge policy-advisor services This will tell you which identities are authorized as clients (Dial privilege) or hosts (Bind privilege).

Just checked. The endpoints used are below. Any issues here?

ctrl.ednpoint (passed to router helmchart) : ztnclient.domain.co:443
ctrlPlane.advertisedHost (in controller): ztnctrl.domain.co
clientApi.advertisedHost (in controller): ztnclient.domain.co

Policy advisor shows router has access to hello service. And hello-cient identity has access to the service aswell.

$ ziti edge policy-advisor services hello-service -q
OKAY : router2 (3) -> hello-service (3) Common Routers: (3/3) Dial: N Bind: Y
OKAY : hello-client (3) -> hello-service (3) Common Routers: (3/3) Dial: Y Bind: N

Router is online

$ ziti edge list edge-routers
╭────────────┬───────────────────┬────────┬───────────────┬──────┬─────────────╮
│ ID         │ NAME              │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES  │
├────────────┼───────────────────┼────────┼───────────────┼──────┼─────────────┤
│ NLQ8jbS-wL │ router2           │ true   │ true          │    0 │             │
╰────────────┴───────────────────┴────────┴───────────────┴──────┴─────────────╯

hello-client identity is added to ziti tunnel service running in local machine.

$ sudo ls /opt/openziti/etc/identities/
hello-client.json

$ sudo cat /opt/openziti/etc/identities/hello-client.json
{
        "ztAPI":"https://ztnclient.domain.co:443",
...
	}

Status of ziti-edge-tunnel service still shows it can't reach the router.

$ sudo systemctl status ziti-edge-tunnel.service 
● ziti-edge-tunnel.service - Ziti Edge Tunnel
     Loaded: loaded (/usr/lib/systemd/system/ziti-edge-tunnel.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-07-16 19:55:31 UTC; 2h 45min ago
    Process: 421518 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
   Main PID: 421521 (ziti-edge-tunne)
      Tasks: 6 (limit: 4586)
     Memory: 5.9M (peak: 6.8M)
        CPU: 1min 12.562s
     CGroup: /system.slice/ziti-edge-tunnel.service
             └─421521 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

Jul 16 22:38:18 ip-172-31-38-32 ziti-edge-tunnel[421521]: (421521)[     9767.202]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[3] failed to connect to ER[router2] [-3001/temporary failure]

Thanks for confirming router2 is online in host mode with bind permission for hello-service. Do you have a default service router policy like this?

❯ ziti edge list service-edge-router-policies
╭────────────────────────┬─────────┬───────────────┬───────────────────╮
│ ID                     │ NAME    │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼─────────┼───────────────┼───────────────────┤
│ 2EFMtpPaQy5MUl47yF544d │ default │ #all          │ #all              │
╰────────────────────────┴─────────┴───────────────┴───────────────────╯
results: 1-1 of 1

If not, does adding one fix it?

ziti edge create service-edge-router-policy "default" \
    --edge-router-roles '#all' --service-roles '#all'

Default router policies do/did exist, but the issue still persist.

$ ziti edge list service-edge-router-policies
╭───────────────────────┬─────────┬───────────────┬───────────────────╮
│ ID                    │ NAME    │ SERVICE ROLES │ EDGE ROUTER ROLES │
├───────────────────────┼─────────┼───────────────┼───────────────────┤
│ mtS0bYousr6TUdoZ2IkUc │ default │ #all          │ #all              │
╰───────────────────────┴─────────┴───────────────┴───────────────────╯
ziti edge list edge-router-policies
╭────────────────────────┬───────────────────────────────┬────────────────────┬────────────────────╮
│ ID                     │ NAME                          │ EDGE ROUTER ROLES  │ IDENTITY ROLES     │
├────────────────────────┼───────────────────────────────┼────────────────────┼────────────────────┤
│ 6l9EaQCeNUyEev5IU18aje │ default                       │ #all               │ #all               │

╰────────────────────────┴───────────────────────────────┴────────────────────┴────────────────────╯

Now I see in the log that the upstream/origin address hello.hello-toy.svc could not be resolved by router2 (the identity with bind privilege). Is router2 a pod in the same cluster where the hello-toy chart was installed? You might be following the K8s service tutorial. You can verify the cluster-internal DNS name of the hello-toy service (the ClusterIP Service's DNS name) by constructing the {cluster service name}.{cluster namespace}.{cluster zone}, so hello.hello-toy.svc is correct if the service name "hello" is deployed in namespace "hello-toy" in cluster zone "svc.cluster.local."

followed the above steps and can confirm

1 - router pod is in same cluster of hello app
2 - hello.hello-toy.svc is resolvable by the pods in cluster
3 - policies are configured

We can replicate same steps and it works on using router service for a linux but issue comes only when we use helm chart router installation .

Using ztnctrl.domain:443 as ctrl endpoint
which is {advertisedHost}:{advertisedPort} format

What we did is we retried the same k3d cluster install provided in guide to check if the domain and port 443 was the issue .

I can confirm that here former works but not latter

This works

helm upgrade --install "ziti-router" openziti/ziti-router \
--namespace "ziti" \
--set-file enrollmentJwt=./router1.jwt \
--set edge.advertisedHost="${NODE_IP}" \
--set edge.advertisedPort=3022 \
--set edge.service.type=LoadBalancer \
--set linkListeners.transport.advertisedHost="${NODE_IP}" \
--set linkListeners.transport.advertisedPort=10080 \
--set linkListeners.transport.service.type=LoadBalancer \
--set tunnel.mode=host \
--set ctrl.endpoint="${NODE_IP}:6262"

This does not work

helm upgrade --install "private-router1" openziti/ziti-router \
--namespace ziti \
--set-file enrollmentJwt=./router1.jwt \
--set edge.advertisedHost=private-router1-edge.ziti.svc.cluster.local \
--set linkListeners.transport.service.enabled=false \
--set tunnel.mode=host \
--set ctrl.endpoint="${NODE_IP}:6262"

I've been using the 2nd one a lot now not sure why we started facing issues with that from past 2 days itself

for our custom domain with 443 port this too works

helm upgrade --install "ziti-router" openziti/ziti-router \
--namespace "ziti" \
--set-file enrollmentJwt=./router1.jwt \
--set edge.advertisedHost="${NODE_IP}" \
--set edge.advertisedPort=3022 \
--set edge.service.type=LoadBalancer \
--set linkListeners.transport.advertisedHost="${NODE_IP}" \
--set linkListeners.transport.advertisedPort=10080 \
--set linkListeners.transport.service.type=LoadBalancer \
--set tunnel.mode=host \
--set ctrl.endpoint="ztnctrl.domain.co:443"

So in summary private router does not work in which ever sort of controller or cluster we try so seems like some breaking change at your side

controllers logs (custom domain one)

{"file":"github.com/openziti/ziti/controller/handler_ctrl/connect.go:117","func":"github.com/openziti/ziti/controller/handler_ctrl.(*ConnectHandler).HandleConnection","level":"error","msg":"unknown/unenrolled router","routerId":"el0zwLEXz","time":"2024-07-17T08:16:31.717Z"}
{"_context":"tls:0.0.0.0:6262","file":"github.com/openziti/channel/v2@v2.0.130/classic_listener.go:201","func":"github.com/openziti/channel/v2.(*classicListener).acceptConnection.func1","level":"error","msg":"connection handler error for [tls:172.31.39.122:34214] (unknown/unenrolled router, routerId: el0zwLEXz)","time":"2024-07-17T08:16:31.717Z"}
{"file":"github.com/openziti/ziti/controller/handler_ctrl/connect.go:117","func":"github.com/openziti/ziti/controller/handler_ctrl.(*ConnectHandler).HandleConnection","level":"error","msg":"unknown/unenrolled router","routerId":"3MheTjS-zL","time":"2024-07-17T08:16:33.274Z"}
{"_context":"tls:0.0.0.0:6262","file":"github.com/openziti/channel/v2@v2.0.130/classic_listener.go:201","func":"github.com/openziti/channel/v2.(*classicListener).acceptConnection.func1","level":"error","msg":"connection handler error for [tls:172.31.39.122:34228] (unknown/unenrolled router, routerId: 3MheTjS-zL)","time":"2024-07-17T08:16:33.274Z"}

Any thoughts @qrkourier as it relates to fix this?

These have to be the same:

{ctrl.ednpoint} == {ctrlPlane.advertisedHost}:{ctrlPlane.advertisedPort}

It looks like your router is configured with ztnclient.domain.co:443, but it should be ztnctrl.domain.co:443.

Explainer:

  • The controller provides an always-mTLS control plane (ctrl) server that's consumed by routers which are configured by ctrl.endpoint.
  • The controller provides a TLS client API for identities and routers to enroll, and for identities to obtain edge sessions with certificate authentication. Both get the URL of the client API from the enrollment token and use the client API's server certificate pubkey to verify the token signature.

Hi, just picking up were they left off.

We've made the changes. But issue still persists. Here is details.

1. Router deployed using terraform-helm

  • ctrl.endpoint=var.ctrlEndpoint=ztnctrl.domain.co:443
resource "helm_release" "router" {
  name       = var.routerName
  repository = "https://openziti.github.io/helm-charts/"
  chart      = "ziti-router"
  version    = "1.0.4"
  namespace = "ziti"
  create_namespace = true

  set {
    name  = "enrollmentJwt"
    value = var.routerjwt
  }

  set {
    name  = "advertisedHost"
    value = format("%s.ziti.svc.cluster.local", var.routerName)
  }

  set {
    name  = "linkListeners.transport.service.enabled"
    value = "false"
  }

  set {
    name  = "ctrl.endpoint"
    value = var.ctrlEndpoint
  }

  set {
    name  = "tunnel.mode"
    value = "host"
  }

  depends_on = [ kubernetes_secret.cloudflare_api_token_secret ]
}
$ kubectl -n ziti get svc
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
demo-router-edge   ClusterIP   10.3.129.152   <none>        443/TCP   7m12s

$ kubectl -n ziti get po
NAME                           READY   STATUS    RESTARTS   AGE
demo-router-6bfd565dcb-mjrhs   1/1     Running   0          7m28s

$ kubectl -n ziti logs demo-router-6bfd565dcb-mjrhs 
{"file":"github.com/openziti/ziti/router/enroll/enroll.go:206","func":"github.com/openziti/ziti/router/enroll.(*RestEnroller).Enroll","level":"info","msg":"registration complete","time":"2024-07-18T02:57:31.684Z"}
{"arch":"amd64","build-date":"2024-05-30T16:36:13Z","configFile":"/etc/ziti/config/ziti-router.yaml","file":"github.com/openziti/ziti/ziti/router/run.go:71","func":"github.com/openziti/ziti/ziti/router.run","go-version":"go1.22.3","level":"info","msg":"starting ziti router","os":"linux","revision":"82c4a7125227","routerId":"1ojTGASXwL","time":"2024-07-18T02:57:31.724Z","version":"v1.1.3"}
{"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":32,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.link.dialer","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":128,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.route.handler","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1,"maxWorkers":50,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.terminator_validation","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:154","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"info","msg":"cached data model file set to: /etc/ziti/config/ziti-router.yaml.json.gzip","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:171","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"warning","msg":"Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:345","func":"github.com/openziti/ziti/router/internal/edgerouter.parseEdgeListenerOptions","level":"info","msg":"advertised port [0] in [listeners[443].options.advertise] does not match the listening port [0] in [listeners[3022].address].","time":"2024-07-18T02:57:31.725Z"}
{"error":"open /etc/ziti/config/ziti-router.yaml.json.gzip: no such file or directory","file":"github.com/openziti/ziti/router/state/manager.go:213","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).LoadRouterModel","level":"error","msg":"could not load router model from file [/etc/ziti/config/ziti-router.yaml.json.gzip]","time":"2024-07-18T02:57:31.725Z"}
{"file":"github.com/openziti/ziti/router/forwarder/faulter.go:78","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"info","msg":"started","time":"2024-07-18T02:57:31.726Z"}
{"file":"github.com/openziti/ziti/router/forwarder/scanner.go:52","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).run","level":"info","msg":"started","time":"2024-07-18T02:57:31.726Z"}
{"file":"github.com/openziti/ziti/router/router.go:346","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"ctrl = {\"OutQueueSize\":4,\"MaxQueuedConnects\":1,\"MaxOutstandingConnects\":16,\"ConnectTimeout\":5000000000,\"DelayRxStart\":false,\"WriteTimeout\":0}","time":"2024-07-18T02:57:31.726Z"}
{"file":"github.com/openziti/ziti/router/router.go:352","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"metrics = {\"ReportInterval\":60000000000,\"IntervalAgeThreshold\":0,\"MessageQueueSize\":10}","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":5000,"maxWorkers":64,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.rate_limiter","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:660","func":"github.com/openziti/ziti/router.(*Router).initializeHealthChecks","level":"info","msg":"starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:481","func":"github.com/openziti/ziti/router.(*Router).startXlinkDialers","level":"info","msg":"started Xlink dialer with binding [transport]","time":"2024-07-18T02:57:31.727Z"}
{"address":{},"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:87","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen","level":"info","msg":"starting channel listener","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1","idleTime":10000000000,"level":"info","maxQueueSize":1,"maxWorkers":16,"minWorkers":1,"msg":"starting goroutine pool","poolType":"pool.listener.xgress_edge","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [edge] at [tls:0.0.0.0:3022]","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [tunnel] at []","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:722","func":"github.com/openziti/ziti/router.(*Router).getInitialCtrlEndpoints","level":"info","msg":"controller endpoints file [/etc/ziti/config/endpoints] doesn't exist. Using initial endpoints from config","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/router.go:555","func":"github.com/openziti/ziti/router.(*Router).startControlPlane","level":"info","msg":"router configured with 1 controller endpoints","time":"2024-07-18T02:57:31.727Z"}
{"endpoint":{"tls:ztnctrl.domain.co:443":{}},"file":"github.com/openziti/ziti/router/env/ctrls.go:93","func":"github.com/openziti/ziti/router/env.(*networkControllers).UpdateControllerEndpoints","level":"info","msg":"adding new ctrl endpoint","time":"2024-07-18T02:57:31.727Z"}
{"endpoint":"tls:ztnctrl.domain.co:443","file":"github.com/openziti/ziti/router/env/ctrls.go:132","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff","level":"info","msg":"starting connection attempts","time":"2024-07-18T02:57:31.727Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge/accept.go:126","func":"github.com/openziti/ziti/router/xgress_edge.(*Acceptor).Run","level":"info","msg":"starting","time":"2024-07-18T02:57:31.727Z"}
{"endpoint":"tls:ztnctrl.domain.co:443","file":"github.com/openziti/ziti/router/env/ctrls.go:138","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func3","level":"info","msg":"successfully connected to controller","time":"2024-07-18T02:57:32.212Z"}
{"ctrlId":"ziti-controller-ctrl-plane-identity","file":"github.com/openziti/ziti/router/link/link_registry.go:306","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).NotifyOfReconnect","level":"info","msg":"resending link states after reconnect","time":"2024-07-18T02:57:32.212Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge/factory.go:76","func":"github.com/openziti/ziti/router/xgress_edge.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-07-18T02:57:32.212Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/factory.go:56","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-07-18T02:57:32.212Z"}
{"file":"github.com/openziti/ziti/router/handler_edge_ctrl/hello.go:82","func":"github.com/openziti/ziti/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1","level":"info","msg":"received server hello, replying","time":"2024-07-18T02:57:32.213Z"}
{"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:203","func":"github.com/openziti/ziti/router/state.(*apiSessionAddedHandler).instantSync","level":"info","msg":"first api session syncId [clyqokwochgxz0d9t7xwk0bc0], starting","strategy":"instant","time":"2024-07-18T02:57:32.338Z"}
{"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:268","func":"github.com/openziti/ziti/router/state.(*apiSessionSyncTracker).Add","level":"info","msg":"received api session sync chunk 0, isLast=true","time":"2024-07-18T02:57:32.339Z"}
{"file":"github.com/openziti/ziti/router/state/manager.go:604","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).StartHeartbeat","level":"info","msg":"heartbeat starting","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/tunneler.go:71","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*tunneler).Start","level":"info","mode":"host","msg":"creating interceptor","time":"2024-07-18T02:57:32.362Z"}
{"error":"exec: \"resolvectl\": executable file not found in $PATH","file":"github.com/openziti/ziti/tunnel/dns/server.go:49","func":"github.com/openziti/ziti/tunnel/dns.flushDnsCaches","level":"warning","msg":"unable to find systemd-resolve or resolvectl in path, consider adding a dns flush to your restart process","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/tunnel/dns/server.go:89","func":"github.com/openziti/ziti/tunnel/dns.NewDnsServer","level":"info","msg":"starting dns server...","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge/certchecker.go:124","func":"github.com/openziti/ziti/router/xgress_edge.(*CertExpirationChecker).Run","level":"info","msg":"waiting 0s to renew certificates","time":"2024-07-18T02:57:32.362Z"}
{"error":"dns server failed to start: listen udp 127.0.0.1:53: bind: permission denied","file":"github.com/openziti/ziti/router/xgress_edge_tunnel/tunneler.go:75","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*tunneler).Start","level":"error","msg":"failed to start DNS resolver. using dummy resolver","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/tunnel/dns/dummy.go:37","func":"github.com/openziti/ziti/tunnel/dns.NewDummyResolver","level":"warning","msg":"dummy resolver does not store hostname/ip mappings","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/tunnel/intercept/iputils.go:51","func":"github.com/openziti/ziti/tunnel/intercept.SetDnsInterceptIpRange","level":"info","msg":"dns intercept IP range: 100.64.0.1 - 100.127.255.255","time":"2024-07-18T02:57:32.362Z"}
{"file":"github.com/openziti/ziti/tunnel/intercept/svcpoll.go:155","func":"github.com/openziti/ziti/tunnel/intercept.(*ServiceListener).HandleServicesChange","level":"info","msg":"adding service","service":"hello-service","time":"2024-07-18T02:57:32.629Z"}
{"file":"github.com/openziti/ziti/tunnel/intercept/svcpoll.go:248","func":"github.com/openziti/ziti/tunnel/intercept.(*ServiceListener).addService","level":"info","msg":"Hosting newly available service","serviceId":"EBDmwwy8tvOBF3iO8BQ09","serviceName":"hello-service","time":"2024-07-18T02:57:32.629Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/fabric.go:396","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*fabricProvider).establishTerminatorWithRetry.func1","level":"info","msg":"attempting to establish terminator","service":"hello-service","time":"2024-07-18T02:57:32.629Z"}
{"channel":"ctrl","file":"github.com/openziti/ziti/router/handler_edge_ctrl/extendEnrollmentCerts.go:126","fingerprint":"90b2c026bc11173c073ac5755be37c0a0f1df3f2","func":"github.com/openziti/ziti/router/handler_edge_ctrl.(*extendEnrollmentCertsHandler).HandleReceive.func1","level":"info","msg":"enrollment extension done","newFingerprint":"76a5874a5d4a1972b7d324bd949ac5efb7bef468","time":"2024-07-18T02:57:32.634Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge/certchecker.go:124","func":"github.com/openziti/ziti/router/xgress_edge.(*CertExpirationChecker).Run","level":"info","msg":"waiting 8591h59m59.3655268s to renew certificates","time":"2024-07-18T02:57:32.634Z"}
{"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/fabric.go:487","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse","level":"info","msg":"received new session","routerId":"1ojTGASXwL","sessionId":"clyqokwzwhgy20d9t41lqzb9j","terminatorId":"4D425VpefAyJaVNK48LzkJ","time":"2024-07-18T02:57:32.759Z"}
{"createDuration":130242501,"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/fabric.go:509","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse","level":"info","msg":"received terminator created notification","routerId":"1ojTGASXwL","terminatorId":"4D425VpefAyJaVNK48LzkJ","time":"2024-07-18T02:57:32.759Z"}
{"file":"github.com/openziti/ziti/router/state/apiSessionAdded.go:124","func":"github.com/openziti/ziti/router/state.(*apiSessionAddedHandler).applySync","level":"info","msg":"finished synchronizing api sessions [count: 6, syncId: clyqokwochgxz0d9t7xwk0bc0, duration: 24.829µs]","time":"2024-07-18T02:57:33.339Z"}
{"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{8Je96}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"4D425VpefAyJaVNK48LzkJ","time":"2024-07-18T02:58:02.335Z"}

2. ziti resources

$ ziti edge list edge-routers
╭────────────┬─────────────┬────────┬───────────────┬──────┬────────────╮
│ ID         │ NAME        │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼─────────────┼────────┼───────────────┼──────┼────────────┤
│ 1ojTGASXwL │ demo-router │ true   │ true          │    0 │            │
╰────────────┴─────────────┴────────┴───────────────┴──────┴────────────╯

$ ziti edge list identities
╭────────────┬───────────────┬─────────┬───────────────┬─────────────╮
│ ID         │ NAME          │ TYPE    │ ATTRIBUTES    │ AUTH-POLICY │
├────────────┼───────────────┼─────────┼───────────────┼─────────────┤
│ 1ojTGASXwL │ demo-router   │ Router  │ hello-hosts   │ Default     │
│ 8jJaPAEXwL │ hello-client  │ Default │ hello-clients │ Default     │
│ gDiEQCdq6  │ Default Admin │ Default │               │ Default     │
╰────────────┴───────────────┴─────────┴───────────────┴─────────────╯


$ ziti edge list configs
╭────────────────────────┬────────────────────────┬──────────────╮
│ ID                     │ NAME                   │ CONFIG TYPE  │
├────────────────────────┼────────────────────────┼──────────────┤
│ 1LD8tmnfFDDWVFAqiK8Pbl │ hello-host-config      │ host.v1      │
│ 2K00XEZssvHN9ZVjACSMZM │ hello-intercept-config │ intercept.v1 │
╰────────────────────────┴────────────────────────┴──────────────╯

$ ziti edge list services
╭───────────────────────┬───────────────┬────────────┬─────────────────────┬────────────╮
│ ID                    │ NAME          │ ENCRYPTION │ TERMINATOR STRATEGY │ ATTRIBUTES │
│                       │               │  REQUIRED  │                     │            │
├───────────────────────┼───────────────┼────────────┼─────────────────────┼────────────┤
│ EBDmwwy8tvOBF3iO8BQ09 │ hello-service │ true       │ smartrouting        │            │
╰───────────────────────┴───────────────┴────────────┴─────────────────────┴────────────╯

$ ziti edge list edge-router-policies
╭───────────────────────┬───────────────────────────────┬───────────────────┬────────────────╮
│ ID                    │ NAME                          │ EDGE ROUTER ROLES │ IDENTITY ROLES │
├───────────────────────┼───────────────────────────────┼───────────────────┼────────────────┤
│ 1ojTGASXwL            │ edge-router-1ojTGASXwL-system │ @demo-router      │ @demo-router   │
│ jzy9aszZwRegEJJyhfGgx │ default                       │ #all              │ #all           │
╰───────────────────────┴───────────────────────────────┴───────────────────┴────────────────╯

# ziti edge list service-edge-router-policies
╭────────────────────────┬─────────┬───────────────┬───────────────────╮
│ ID                     │ NAME    │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼─────────┼───────────────┼───────────────────┤
│ 4bNtqqgBodS3Ybz8fGgn2b │ default │ #all          │ #all              │
╰────────────────────────┴─────────┴───────────────┴───────────────────╯

$ ziti edge list service-policies
╭────────────────────────┬───────────────────┬──────────┬────────────────┬────────────────┬─────────────────────╮
│ ID                     │ NAME              │ SEMANTIC │ SERVICE ROLES  │ IDENTITY ROLES │ POSTURE CHECK ROLES │
├────────────────────────┼───────────────────┼──────────┼────────────────┼────────────────┼─────────────────────┤
│ 1VVFRSdie6Ox7CZRJ0CavZ │ hello-dial-policy │ AllOf    │ @hello-service │ #hello-clients │                     │
│ S4MsBqv8QHLL2BwzXUiD1  │ hello-bind-policy │ AllOf    │ @hello-service │ #hello-hosts   │                     │
╰────────────────────────┴───────────────────┴──────────┴────────────────┴────────────────┴─────────────────────╯

3. Testing Connection

$ ziti edge policy-advisor services hello-service -q
OKAY : demo-router (1) -> hello-service (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : hello-client (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N

$ ls /opt/openziti/etc/identities/
hello-client.json

$ systemctl status ziti-edge-tunnel.service
● ziti-edge-tunnel.service - Ziti Edge Tunnel
     Loaded: loaded (/usr/lib/systemd/system/ziti-edge-tunnel.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-07-18 02:39:12 UTC; 36min ago
    Process: 526778 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
   Main PID: 526784 (ziti-edge-tunne)
      Tasks: 6 (limit: 4586)
     Memory: 5.5M (peak: 6.7M)
        CPU: 11.734s
     CGroup: /system.slice/ziti-edge-tunnel.service
             └─526784 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

Jul 18 03:04:51 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1538.985]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:04:57 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1544.792]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:07:11 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1678.786]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:08:00 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1727.559]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:10:01 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1848.177]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:11:55 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1962.089]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:12:18 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1985.287]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:12:24 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     1991.357]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:14:18 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     2105.765]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>
Jul 18 03:15:29 ip-172-31-38-32 ziti-edge-tunnel[526784]: (526784)[     2176.490]   ERROR ziti-sdk:channel.c:903 on_channel_connect_internal() ch[1] failed to connect to ER[demo-router] [-3001/temporary failu>

$ curl -v hello.ziti.internal
* Host hello.ziti.internal:80 was resolved.
* IPv6: (none)
* IPv4: 100.64.0.3
*   Trying 100.64.0.3:80...
* connect to 100.64.0.3 port 80 from 100.64.0.1 port 52150 failed: Connection timed out
* Failed to connect to hello.ziti.internal port 80 after 134826 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to hello.ziti.internal port 80 after 134826 ms: Couldn't connect to server

@qrkourier do you think it would help to solve this via live call if possible? This is on critical path for us and any help is most appreciated

Thank you for that offer. I'm working to reproduce it now.

I see at least one chart version 1.0.4 was apparently triggering the issue. Is the same reproducible with latest router chart version 1.0.7?

I ran through this tutorial just now with latest charts and did not encounter the issue.

I'll join this Jitsi Meet in a few minutes in case you can huddle.

I think we tried the latest one too but still to confirm will try again and update if meet is needed . thanks alot . Also the reason we used that specific version was because in OVH cluster we have issues related to syntax when we deploy

confirmed , it does not work

private-router1	ziti     	1       	2024-07-18 13:18:36.556543915 +0000 UTC	deployed	ziti-router-1.0.7     	1.1.3  

Sadath's issue here was that there were no reachable edge listeners ({edge.advertisedHost}:{edge.advertisedPort}) from the client's perspective.

I'm still working to clarify and reproduce @bazooka720 and @yemaney's issues.

{"file":"github.com/openziti/ziti/router/router.go:346","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"ctrl = {\"OutQueueSize\":4,\"MaxQueuedConnects\":1,\"MaxOutstandingConnects\":16,\"ConnectTimeout\":5000000000,\"DelayRxStart\":false,\"WriteTimeout\":0}","time":"2024-07-18T02:57:31.726Z"}

@yemaney Your last router log showed a ctrl connection error info message. That means the router failed to connect to the controller's {ctrlPlane.advertisedHost}:{ctrlPlane.advertisedPort}. That's an mTLS connection, so if you're advertising port 443/tcp through a load balancer, then ensure it's configured for passthrough TLS (a layer 4 TCP transport proxy, optionally selecting upstream/origin by SNI).

Now I see it's an INFO message, not an ERROR, so I don't think it's a connectivity problem from router to ctrl.

@yemaney Let's check the log from your client that got "connection refused," despite being authorized to dial the service. If you're using the tunneler package on Linux then you can capture the log from the current run, assuming you reproduced it this tunneler run, with:

journalctl _SYSTEMD_INVOCATION_ID=$(systemctl show -p InvocationID --value ziti-edge-tunnel.service) -l --no-pager

Look for error messages at the same time you attempt to connect. You can follow that log in realtime too.

journalctl -lfu ziti-edge-tunnel.service

If you don't see any interesting messages you can bump up the level to DEBUG.

sudo ziti-edge-tunnel set_log_level --loglevel DEBUG

@bazooka720 Is this correct? The client tunneler can't reach the router then it's running in K8s, but it can reach it when the router package is installed in Linux. When the router is in K8s, the client logs "failed to connect to ER."

You may be experiencing the same issue that Sadath had where the client has no reachable edge listeners.

The policy advisor can only show what is authorized, so we need to double check that the dialer and binder both can reach at least one edge listener. It doesn't have to be the same edge listener for both, but they must have at least one online router in common.

To simplify things, let's say there's only one router. It must have an edge listener (advertisedHost:advertisedPort) that is reachable by the client. We know the client API is reachable because we have already enrolled successfully. We know the binder (hosting tunneler) can reach the target server. We know the router can reach the ctrlPlane host:port because it shows "online" status when listed with the CLI, correct? There's no question about link listeners until we have more than one router. That's how they build the mesh fabric. There's only one router in this scenario, so no inter-router links.