Connect timeout: no suitable edge router

Support
1

I am using the SaaS CloudZiti service to get to know OpenZiti better.

At the moment i am trying to set up a basic network with one service (a Home Assistant (HA) web ui) bein available for my identities (windows pc and adnroid phone).

Unfortunately, I can't get it working to access to HA web ui. I can ping the service client address (homeassistant.csc:80) from my pc with an enrolled identity but I can't curl or open the web ui in the browser.

The ziti-tunneler.log on the PC shows following errors concerning the edge router:

[2024-02-15T11:25:23.158Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[Home Assistant Web GUI]
[2024-02-15T11:25:23.158Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1263 on_event() =============== service event (added) - Home Assistant Web GUI:3U12jw2Ctdb2j7b56DNc72 ===============
[2024-02-15T11:25:23.677Z]    INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
[2024-02-15T11:25:46.187Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.3] for query[1:homeassistant.csc]
[2024-02-15T11:25:56.498Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.0/Connecting] connect timeout: no suitable edge router
[2024-02-15T11:25:56.498Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-15T11:25:56.605Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.1/Connecting] connect timeout: no suitable edge router
[2024-02-15T11:25:56.605Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-15T11:25:56.699Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.2/Connecting] connect timeout: no suitable edge router
...

the service is accessed by netfoundry openziti home assistant addon. the log of the add on shows some weird behavior, even though the identity seems to be sucessfully enrolled on the addon:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/profile.sh
cont-init: info: /etc/cont-init.d/profile.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun ziti-edge-tunnel (no readiness notification)
s6-rc: info: service legacy-services successfully started
[12:47:48] NOTICE: Starting Ziti-Edge-Tunnel...
[12:47:48] INFO:   with SUPERVISOR_TOKEN  "505529ea29...".
[12:47:48] INFO:   with IdentityDirectory "/share/openziti/identities".
[12:47:48] INFO:   with ResolutionRange   "100.64.64.0/18".
[12:47:48] INFO:   with UpstreamResolver  "192.168.128.1".
[12:47:48] INFO:   with LogLevel          "2".
[12:47:48] NOTICE: ZITI-EDGE-TUNNEL: PREINIT BEGIN
[12:47:48] INFO: ZITI-EDGE-TUNNEL: ENROLLMENT NOT REQUESTED
[12:47:48] INFO: IDENTITY: [/share/openziti/identities/ZTID-20240215_114559.json]
[12:47:48] INFO: ZITI_DNS_IP: 100.64.64.1
[12:47:48] INFO: Setup of system resolver via REST to [100.64.64.1] succeeded.
[12:47:48] INFO: Assisting application "nginx" has been started with syntax options "NONE".
[12:47:48] INFO: Assisting application "php-fpm82" has been started with syntax options "NONE".
[12:47:48] INFO: INIT STRING: [/opt/openziti/ziti-edge-tunnel run -I /share/openziti/identities -d 100.64.64.0/18 -u 192.168.128.1 -v 2]
[12:47:48] NOTICE: ZITI-EDGE-TUNNEL: PREINIT END
[12:47:48] NOTICE: ZITI-EDGE-TUNNEL: PROGRAM BEGIN
(180)[        0.000]   ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json. This is normal if this is a new install or if the config file was removed manually
(180)[        0.000]   ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json.backup. This is normal if this is a new install or if the config file was removed manually
(180)[        0.000]    WARN ziti-edge-tunnel:instance-config.c:98 load_tunnel_status_from_file() Config files /var/lib/ziti/config.json and the backup file cannot be read or they do not exist, will create a new config file or the old one will be overwritten
(180)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/share/openziti/identities/ZTID-20240215_114559.json] is not loaded yet or already removed.
(180)[        0.041]    WARN ziti-edge-tunnel:tun.c:277 find_dns_updater() Adding ziti resolver to /etc/resolv.conf. Ziti DNS functionality may be impaired
(180)[        0.043]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
(180)[        5.058]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io] request failed: -3001(temporary failure)
(180)[        5.058]   ERROR ziti-sdk:ziti.c:1668 version_cb() ztx[0] failed to get controller version from https://425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io:443 CONTROLLER_UNAVAILABLE(temporary failure)
(180)[        5.058]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io] request failed: -3001(temporary failure)
(180)[        5.058]    WARN ziti-sdk:ziti.c:1597 api_session_cb() ztx[0] failed to get api session from ctrl[https://425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io:443] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] temporary failure
(180)[        5.058]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:781 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
(180)[        5.058]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1202 on_event() ztx[/share/openziti/identities/ZTID-20240215_114559.json] failed to connect to controller due to ziti controller is not available
(180)[       78.178]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io] request failed: -110(connection timed out)
(180)[       78.178]   ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out
(180)[       78.178]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io] request failed: -110(connection timed out)
(180)[       78.178]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io] request failed: -110(connection timed out)
(180)[       78.178]   ERROR ziti-sdk:ziti.c:1099 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io:443]
(180)[       78.178]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:781 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
(180)[       78.178]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1202 on_event() ztx[/share/openziti/identities/ZTID-20240215_114559.json] failed to connect to controller due to ziti controller is not available
(180)[      146.175]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
(180)[      146.175]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0

Following you find my config from the Netfoundry CloudZiti Ui:

The identity for my PC (client) ist enrolled like this:

Summary 
{
  "model": {
    "id": "4e287348-813d-45be-8bab-7dcec3540e0a",
    "ownerIdentityId": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdBy": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdAt": "2024-02-15T09:31:15.966413Z",
    "updatedAt": "2024-02-15T11:25:28.738142Z",
    "deletedBy": null,
    "deletedAt": null,
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "zitiId": "1hAZGfy8RI",
    "name": "Mr-PC",
    "typeId": "Default",
    "appId": "Ziti Desktop Edge for Windows",
    "appVersion": "2.1.16",
    "branch": "HEAD",
    "revision": "68c3a76",
    "type": "ziti-sdk-c",
    "version": "0.31.4",
    "arch": "x86_64",
    "os": "MINGW32_NT-10.0",
    "osRelease": "10.0.22631",
    "osVersion": "Windows 10 Pro",
    "externalId": null,
    "authPolicyId": "d5941f80-056c-4f16-87f1-d486f3755b50",
    "disabled": false,
    "disabledAt": null,
    "disabledUntil": null,
    "hasApiSession": true,
    "hasEdgeRouterConnection": false,
    "lastOnlineAt": null,
    "syncId": null,
    "syncResourceId": null,
    "attributes": [
      "#user-device"
    ],
    "interceptConflicts": [],
    "loopbackConflicts": [],
    "online": false,
    "managedBy": "UNMANAGED",
    "jwtExpiresAt": null,
    "mfaEnabled": false,
    "jwt": null,
    "_meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:52:03.079940510Z"
      },
      "ziti": {
        "id": "1hAZGfy8RI",
        "name": "Mr-PC",
        "type": {
          "id": "Default",
          "name": "Default"
        },
        "typeId": "Default",
        "externalId": null,
        "authPolicyId": "default",
        "authenticators": {
          "updb": null,
          "cert": {
            "fingerprint": "b0313d04a97dfcdf9ebe299232077cdd71d166f8"
          }
        },
        "enrollment": {
          "ott": null,
          "ottca": null,
          "updb": null
        },
        "disabled": false,
        "disabledAt": null,
        "disabledUntil": null,
        "envInfo": {
          "arch": "x86_64",
          "os": "MINGW32_NT-10.0",
          "osRelease": "10.0.22631",
          "osVersion": "Windows 10 Pro"
        },
        "sdkInfo": {
          "appId": "Ziti Desktop Edge for Windows",
          "appVersion": "2.1.16",
          "branch": "HEAD",
          "revision": "68c3a76",
          "type": "ziti-sdk-c",
          "version": "0.31.4"
        },
        "hasApiSession": true,
        "hasEdgeRouterConnection": false,
        "roleAttributes": [
          "user-device"
        ],
        "tags": {},
        "appData": {},
        "createdAt": "2024-02-15T09:31:16.117Z",
        "updatedAt": "2024-02-15T11:21:32.826Z",
        "admin": false,
        "defaultAdmin": false,
        "mfaEnabled": false
      }
    },
    "meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:52:03.079940510Z"
      },
      "ziti": {
        "id": "1hAZGfy8RI",
        "name": "Mr-PC",
        "type": {
          "id": "Default",
          "name": "Default"
        },
        "typeId": "Default",
        "externalId": null,
        "authPolicyId": "default",
        "authenticators": {
          "updb": null,
          "cert": {
            "fingerprint": "b0313d04a97dfcdf9ebe299232077cdd71d166f8"
          }
        },
        "enrollment": {
          "ott": null,
          "ottca": null,
          "updb": null
        },
        "disabled": false,
        "disabledAt": null,
        "disabledUntil": null,
        "envInfo": {
          "arch": "x86_64",
          "os": "MINGW32_NT-10.0",
          "osRelease": "10.0.22631",
          "osVersion": "Windows 10 Pro"
        },
        "sdkInfo": {
          "appId": "Ziti Desktop Edge for Windows",
          "appVersion": "2.1.16",
          "branch": "HEAD",
          "revision": "68c3a76",
          "type": "ziti-sdk-c",
          "version": "0.31.4"
        },
        "hasApiSession": true,
        "hasEdgeRouterConnection": false,
        "roleAttributes": [
          "user-device"
        ],
        "tags": {},
        "appData": {},
        "createdAt": "2024-02-15T09:31:16.117Z",
        "updatedAt": "2024-02-15T11:21:32.826Z",
        "admin": false,
        "defaultAdmin": false,
        "mfaEnabled": false
      }
    }
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}

The identity for service access (enrolled on the netfoundry openziti home assistant addon) looks like this:

Summary 
{
  "model": {
    "id": "9e1ff1f3-e501-4af3-8581-e77a8c4397de",
    "ownerIdentityId": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdBy": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdAt": "2024-02-15T09:25:38.702691Z",
    "updatedAt": "2024-02-15T10:46:24.898984Z",
    "deletedBy": null,
    "deletedAt": null,
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "zitiId": "qzjkXhVo0",
    "name": "Home Assistant",
    "typeId": "Default",
    "appId": "ziti-edge-tunnel",
    "appVersion": "0.22.20",
    "branch": "HEAD",
    "revision": "5acfb13",
    "type": "ziti-sdk-c",
    "version": "0.35.12",
    "arch": "aarch64",
    "os": "Linux",
    "osRelease": "6.1.63-haos-raspi",
    "osVersion": "#1 SMP PREEMPT Mon Feb  5 11:43:32 UTC 2024",
    "externalId": null,
    "authPolicyId": "d5941f80-056c-4f16-87f1-d486f3755b50",
    "disabled": false,
    "disabledAt": null,
    "disabledUntil": null,
    "hasApiSession": true,
    "hasEdgeRouterConnection": false,
    "lastOnlineAt": null,
    "syncId": null,
    "syncResourceId": null,
    "attributes": [
      "#service-host"
    ],
    "interceptConflicts": [],
    "loopbackConflicts": [],
    "online": false,
    "managedBy": "UNMANAGED",
    "jwtExpiresAt": null,
    "mfaEnabled": false,
    "jwt": null,
    "_meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:52:36.173751253Z"
      },
      "ziti": {
        "id": "qzjkXhVo0",
        "name": "Home Assistant",
        "type": {
          "id": "Default",
          "name": "Default"
        },
        "typeId": "Default",
        "externalId": null,
        "authPolicyId": "default",
        "authenticators": {
          "updb": null,
          "cert": {
            "fingerprint": "ef5d888e1fcd553ece3401b0d5c58120507dfad1"
          }
        },
        "enrollment": {
          "ott": null,
          "ottca": null,
          "updb": null
        },
        "disabled": false,
        "disabledAt": null,
        "disabledUntil": null,
        "envInfo": {
          "arch": "aarch64",
          "os": "Linux",
          "osRelease": "6.1.63-haos-raspi",
          "osVersion": "#1 SMP PREEMPT Mon Feb  5 11:43:32 UTC 2024"
        },
        "sdkInfo": {
          "appId": "ziti-edge-tunnel",
          "appVersion": "0.22.20",
          "branch": "HEAD",
          "revision": "5acfb13",
          "type": "ziti-sdk-c",
          "version": "0.35.12"
        },
        "hasApiSession": true,
        "hasEdgeRouterConnection": false,
        "roleAttributes": [
          "service-host"
        ],
        "tags": {},
        "appData": {},
        "createdAt": "2024-02-15T09:25:38.797Z",
        "updatedAt": "2024-02-15T09:31:42.418Z",
        "admin": false,
        "defaultAdmin": false,
        "mfaEnabled": false
      }
    },
    "meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:52:36.173751253Z"
      },
      "ziti": {
        "id": "qzjkXhVo0",
        "name": "Home Assistant",
        "type": {
          "id": "Default",
          "name": "Default"
        },
        "typeId": "Default",
        "externalId": null,
        "authPolicyId": "default",
        "authenticators": {
          "updb": null,
          "cert": {
            "fingerprint": "ef5d888e1fcd553ece3401b0d5c58120507dfad1"
          }
        },
        "enrollment": {
          "ott": null,
          "ottca": null,
          "updb": null
        },
        "disabled": false,
        "disabledAt": null,
        "disabledUntil": null,
        "envInfo": {
          "arch": "aarch64",
          "os": "Linux",
          "osRelease": "6.1.63-haos-raspi",
          "osVersion": "#1 SMP PREEMPT Mon Feb  5 11:43:32 UTC 2024"
        },
        "sdkInfo": {
          "appId": "ziti-edge-tunnel",
          "appVersion": "0.22.20",
          "branch": "HEAD",
          "revision": "5acfb13",
          "type": "ziti-sdk-c",
          "version": "0.35.12"
        },
        "hasApiSession": true,
        "hasEdgeRouterConnection": false,
        "roleAttributes": [
          "service-host"
        ],
        "tags": {},
        "appData": {},
        "createdAt": "2024-02-15T09:25:38.797Z",
        "updatedAt": "2024-02-15T09:31:42.418Z",
        "admin": false,
        "defaultAdmin": false,
        "mfaEnabled": false
      }
    }
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}

I have two netfroundry managed edge routers up and runnning:

grafik
grafik1454×175 20.1 KB

the service for accessing the home assistant web ui look like this:

Summary 
{
  "model": {
    "id": "2229dd3a-fda2-4848-a969-2d34e798c122",
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "zitiId": "3U12jw2Ctdb2j7b56DNc72",
    "name": "Home Assistant Web GUI",
    "encryptionRequired": true,
    "attributes": [],
    "modelType": "TunnelerToEndpoint",
    "ownerIdentityId": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdBy": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdAt": "2024-02-15T09:44:28.281098Z",
    "updatedAt": "2024-02-15T09:44:28.532058Z",
    "deletedBy": null,
    "deletedAt": null,
    "configIdByConfigTypeId": {
      "a069310a-e060-4de5-b779-2ae25ca2488d": "7517798a-cb2c-494d-a6c9-49e0ca869fb8",
      "fac8d628-31e9-4ee5-886c-663d0eaca15b": "c97ff280-91c0-4a78-ab81-9d38cd6f32f0"
    },
    "model": {
      "clientIngress": {
        "host": "homeassistant.csc",
        "port": 80
      },
      "serverEgress": {
        "protocol": "tcp",
        "host": "192.168.128.23",
        "port": 8123
      },
      "bindEndpointAttributes": [
        "@Home Assistant"
      ],
      "edgeRouterAttributes": []
    },
    "_meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T12:05:38.785914808Z"
      },
      "ziti": {
        "id": "3U12jw2Ctdb2j7b56DNc72",
        "name": "Home Assistant Web GUI",
        "encryptionRequired": true,
        "terminatorStrategy": "smartrouting",
        "roleAttributes": null,
        "configs": [
          "28rKFFERRqsypfyQvqsFFU",
          "4xhlEcLajJA7dfHz5r1pT9"
        ],
        "tags": {
          "network-id": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
          "resource-id": "2229dd3a-fda2-4848-a969-2d34e798c122"
        },
        "createdAt": "2024-02-15T09:44:28.524Z",
        "updatedAt": "2024-02-15T10:26:52.986Z"
      }
    },
    "meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T12:05:38.785914808Z"
      },
      "ziti": {
        "id": "3U12jw2Ctdb2j7b56DNc72",
        "name": "Home Assistant Web GUI",
        "encryptionRequired": true,
        "terminatorStrategy": "smartrouting",
        "roleAttributes": null,
        "configs": [
          "28rKFFERRqsypfyQvqsFFU",
          "4xhlEcLajJA7dfHz5r1pT9"
        ],
        "tags": {
          "network-id": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
          "resource-id": "2229dd3a-fda2-4848-a969-2d34e798c122"
        },
        "createdAt": "2024-02-15T09:44:28.524Z",
        "updatedAt": "2024-02-15T10:26:52.986Z"
      }
    }
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}

the service policy:

Summary 
{
  "model": {
    "id": "bbda3e30-46f4-46b3-9c41-bd7fe21fcbfd",
    "ownerIdentityId": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdBy": "c5cf88e0-5d14-4523-9dde-d3454e42792c",
    "createdAt": "2024-02-15T09:46:39.640658Z",
    "updatedAt": "2024-02-15T09:46:39.722803Z",
    "deletedBy": null,
    "deletedAt": null,
    "name": "Access Home Assistant Web GUI",
    "zitiId": "2twBWr63swuFZflZFsnb1W",
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "serviceAttributes": [
      "#all"
    ],
    "endpointAttributes": [
      "#all"
    ],
    "postureCheckAttributes": [],
    "interceptConflicts": [],
    "_meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:53:53.567788797Z"
      },
      "ziti": {
        "id": "2twBWr63swuFZflZFsnb1W",
        "name": "Access Home Assistant Web GUI",
        "type": "Dial",
        "semantic": "AnyOf",
        "identityRoles": [
          "#all"
        ],
        "serviceRoles": [
          "#all"
        ],
        "postureCheckRoles": null,
        "tags": {
          "network-id": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
          "resource-id": "bbda3e30-46f4-46b3-9c41-bd7fe21fcbfd"
        },
        "createdAt": "2024-02-15T09:46:39.716Z",
        "updatedAt": "2024-02-15T11:28:14.817Z"
      }
    },
    "meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:53:53.567788797Z"
      },
      "ziti": {
        "id": "2twBWr63swuFZflZFsnb1W",
        "name": "Access Home Assistant Web GUI",
        "type": "Dial",
        "semantic": "AnyOf",
        "identityRoles": [
          "#all"
        ],
        "serviceRoles": [
          "#all"
        ],
        "postureCheckRoles": null,
        "tags": {
          "network-id": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
          "resource-id": "bbda3e30-46f4-46b3-9c41-bd7fe21fcbfd"
        },
        "createdAt": "2024-02-15T09:46:39.716Z",
        "updatedAt": "2024-02-15T11:28:14.817Z"
      }
    }
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}

the edge router policy:

Summary 
{
  "model": {
    "id": "aa0b4b2b-e3e8-4ddd-af4e-dcc0418b8e9f",
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "zitiId": "4eQ6c1QksX6aqdlVZpHMMv",
    "name": "Default Edge Router Policy Fabric/All",
    "isSystem": false,
    "semantic": "AnyOf",
    "endpointAttributes": [
      "#all"
    ],
    "edgeRouterAttributes": [
      "#fabric"
    ],
    "exclusiveType": null,
    "ownerIdentityId": "26690eb4-92e9-4ed0-acbe-6e26564eae99",
    "createdBy": "26690eb4-92e9-4ed0-acbe-6e26564eae99",
    "createdAt": "2024-02-07T17:43:34.299786Z",
    "updatedAt": "2024-02-15T08:56:29.972139Z",
    "deletedBy": null,
    "deletedAt": null,
    "_meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:54:26.391728416Z"
      },
      "ziti": {
        "id": "4eQ6c1QksX6aqdlVZpHMMv",
        "name": "Default Edge Router Policy Fabric/All",
        "tags": {},
        "semantic": "AnyOf",
        "isSystem": false,
        "edgeRouterRoles": [
          "#fabric"
        ],
        "identityRoles": [
          "#all"
        ],
        "createdAt": "2024-02-07T17:43:33.288Z",
        "updatedAt": "2024-02-15T11:00:32.584Z"
      }
    },
    "meta": {
      "diffZiti": {
        "differencesDetected": false,
        "description": "No differences detected",
        "comparedAt": "2024-02-15T11:54:26.391728416Z"
      },
      "ziti": {
        "id": "4eQ6c1QksX6aqdlVZpHMMv",
        "name": "Default Edge Router Policy Fabric/All",
        "tags": {},
        "semantic": "AnyOf",
        "isSystem": false,
        "edgeRouterRoles": [
          "#fabric"
        ],
        "identityRoles": [
          "#all"
        ],
        "createdAt": "2024-02-07T17:43:33.288Z",
        "updatedAt": "2024-02-15T11:00:32.584Z"
      }
    }
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}

the service edge router policy:

Summary 
{
  "model": {
    "id": "225bbed6-78f3-45d0-a325-da905c86201c",
    "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
    "zitiId": "2trQZJ5nkBti1t4fQtTTDl",
    "name": "Service Edge Router Policy All/All",
    "semantic": "AnyOf",
    "serviceAttributes": [
      "#all"
    ],
    "edgeRouterAttributes": [
      "#all"
    ],
    "exclusiveType": null,
    "ownerIdentityId": "26690eb4-92e9-4ed0-acbe-6e26564eae99",
    "createdBy": "26690eb4-92e9-4ed0-acbe-6e26564eae99",
    "createdAt": "2024-02-07T17:43:35.035849Z",
    "updatedAt": "2024-02-07T17:43:35.093893Z",
    "deletedBy": null,
    "deletedAt": null,
    "_links": {
      "self": {
        "href": "https://gateway.production.netfoundry.io/core/v2/service-edge-router-policies/225bbed6-78f3-45d0-a325-da905c86201c",
        "title": "Service Edge Router Policy All/All"
      },
      "network": {
        "href": "https://gateway.production.netfoundry.io/core/v2/networks/ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
        "title": "finished-nap-speech-glass",
        "profile": "parent"
      }
    },
    "actionList": [
      "update",
      "delete",
      "json"
    ]
  },
  "resourceType": "endpoint",
  "networkId": "ca678c8a-8ab3-4f78-b2b6-0bbcc4136053",
  "networkGroupId": "79cc69b9-77da-430e-b09f-79470e630be5"
}
2

Hi @MrRatherford, welcome to the community, to OpenZiti and CloudZiti!

Thanks for all those details. This definitely seems to be "the issue" and it points to the device/identity not being able to connect to a router...

connect timeout: no suitable edge router

"No suitable edge router" usually means the router you have access to are all offline, or you don't have any routers available to that identity, or there's a severe network issue impacting connectivity to the router.

Based on the router policy you shared I can see that any router with the #fabric attribute should be usable by #all (any) identities. That should definitely have you covered...

The connection to the controller messages are certainly troubling. This one seems relevant:

(180)[       78.178]   ERROR ziti-sdk:ziti.c:1308 edge_routers_cb() ztx[0] failed to get current edge routers: CONTROLLER_UNAVAILABLE/connection timed out

That sure looks to me like the source of the problem... Are you able to connect to that url? If you run this command:

openssl s_client -connect 425e8315-9015-4d57-aaba-4f2bb9874f67.production.netfoundry.io:443 -showcerts < /dev/null

do you get a response in a timely fashion?

3

Hi @MrRatherford - Indeed, welcome to the community!

Thanks for checking out the HA integration of OpenZiti. I noticed a change in the way Ziti was handling DNS and it seems like this might be related to your issue. To remedy until I can really research the best method, I have placed a fix in the latest version of the integration. You should see a new version in the repo now (1.5). Give that a shot and let me know if it can connect afterwards. Thanks a bunch!

4

Thanks for the quick replies! We had an Internet outage caused by Our Provider therefore I am a little late with my answer.

@TheLumberjack: i ran the command and the results for the netfoundry Servers are showing certificates and no errors. The Client side looks like this:

1000067226
10000672261080×2340 183 KB

Could the self-signed certificate be the problem?

@NicFragale: concluded from the openssl command the URL of the netfoundry Server could be resolved.

After updating the ziti Homeassistant addon to 1.5.1 I get the following logs after startup:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/profile.sh
cont-init: info: /etc/cont-init.d/profile.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun ziti-edge-tunnel (no readiness notification)
s6-rc: info: service legacy-services successfully started
[10:42:12] NOTICE: Starting Ziti-Edge-Tunnel...
[10:42:12] INFO: with SUPERVISOR_TOKEN  :"cb0c1cce6a...".
[10:42:12] INFO: with IdentityDirectory :"/share/openziti/identities".
[10:42:12] INFO: with ResolutionRange   :"100.64.64.0/18".
[10:42:12] INFO: with UpstreamResolver  :"192.168.128.1".
[10:42:12] INFO: with LogLevel          :"2".
[10:42:12] NOTICE: ZITI-EDGE-TUNNEL: PREINIT BEGIN
[10:42:12] INFO: Runtime version is "0.22.19".
[10:42:12] INFO: Architecture is "aarch64".
[10:42:12] INFO: ZITI-EDGE-TUNNEL: ENROLLMENT NOT REQUESTED
[10:42:12] INFO: IDENTITY: [/share/openziti/identities/ZTID-20240215_114559.json]
[10:42:12] INFO: ZITI_DNS_IP: 100.64.64.1
[10:42:12] INFO: Setup of system resolver via REST to [100.64.64.1] succeeded.
[10:42:12] INFO: Assisting application "nginx" has been started with syntax options "NONE".
[10:42:12] INFO: Assisting application "php-fpm82" has been started with syntax options "NONE".
[10:42:13] INFO: INIT STRING: [/opt/openziti/ziti-edge-tunnel run -I /share/openziti/identities -u 192.168.128.1 -v 2]
[10:42:13] NOTICE: ZITI-EDGE-TUNNEL: PREINIT END
[10:42:13] NOTICE: ZITI-EDGE-TUNNEL: PROGRAM BEGIN
(184)[        0.000]   ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json. This is normal if this is a new install or if the config file was removed manually
(184)[        0.000]   ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json.backup. This is normal if this is a new install or if the config file was removed manually
(184)[        0.000]    WARN ziti-edge-tunnel:instance-config.c:98 load_tunnel_status_from_file() Config files /var/lib/ziti/config.json and the backup file cannot be read or they do not exist, will create a new config file or the old one will be overwritten
(184)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/share/openziti/identities/ZTID-20240215_114559.json] is not loaded yet or already removed.
(184)[        0.077]    WARN ziti-edge-tunnel:tun.c:277 find_dns_updater() Adding ziti resolver to /etc/resolv.conf. Ziti DNS functionality may be impaired
(184)[        0.077]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
[10:42:18] INFO: UPDATED RESOLV CONFIGURATION
[10:47:08] INFO: ZITI-EDGE-TUNNEL: [1/Sat Feb 17 10:47:08 CET 2024] [PID:184] [WAIT:MAIN LOOP]
[10:52:09] INFO: ZITI-EDGE-TUNNEL: [2/Sat Feb 17 10:52:09 CET 2024] [PID:184] [WAIT:MAIN LOOP]
[10:57:09] INFO: ZITI-EDGE-TUNNEL: [3/Sat Feb 17 10:57:09 CET 2024] [PID:184] [WAIT:MAIN LOOP]

When i try to connect from android the ziti App logs again: EdgeRouterUnavailable and the logs on the homeassistant addon site dont Show an connection attempt.

5

No. OpenZiti maintains its own PKI and establishes trust outside of the operating system, so certs signed by OpenZiti's controller and not an independent 3rd party are fine and to be expected.

The ziti edge tunnel/homeassistant error Nic will have to weigh in on but the android app stating "EdgeRouterUnavailable" is still the most troubling problem to me.

Have you been able to use your network for anything yet or is this the first thing and you're hitting this issue? It might be worthwhile to just make sure it's healthy overall by doing something trivial? I could help you with that if you want to go that route.

6

I put up some test services which are hosted by different identities (one on android deployed in the ziti android app, the other one on a Windows PC running the official ziti windows client) in the ziti network. Whenever I tried to reach them from another device with another identity the connection timed out:

[2024-02-17T20:10:31.057Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:31.152Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:31.261Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:41.530Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:41.633Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:41.728Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:52.259Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:52.352Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.7] for query[1:homeassistant.csc]
[2024-02-17T20:10:52.352Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:52.353Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.7] for query[1:homeassistant.csc]
[2024-02-17T20:10:52.463Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:10:59.524Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.8] for query[1:website2.csc]
[2024-02-17T20:10:59.524Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.8] for query[1:website2.csc]
[2024-02-17T20:11:02.353Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.7/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:02.353Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:02.613Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.8/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:02.613Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:02.733Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:02.828Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:02.919Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:10.011Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.9/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:10.011Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:10.117Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.10/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:10.117Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:13.048Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:13.144Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:13.251Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:20.525Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.11/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:20.525Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:20.635Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.12/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:11:20.635Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:11:23.533Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:23.644Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:23.738Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:34.011Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:11:34.110Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:08.773Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:08.882Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:08.977Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:11.521Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.6] for query[1:website.csc]
[2024-02-17T20:13:11.521Z]    INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.6] for query[1:website.csc]
[2024-02-17T20:13:19.238Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.2/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:19.347Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.0/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:19.442Z]    WARN ziti-sdk:connect.c:356 connect_timeout() conn[0.1/Binding] bind timeout: no suitable edge router
[2024-02-17T20:13:21.839Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.13/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:13:21.839Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time
[2024-02-17T20:13:21.949Z]    WARN ziti-sdk:connect.c:348 connect_timeout() conn[0.14/Connecting] connect timeout: no suitable edge router
[2024-02-17T20:13:21.949Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: Operation did not complete in time

The last connection attempt was from my Windows PC to a service itself hosted. I don't know if this would be possible if everything works fine, but I just tried :wink:

Therfore I think it would be best to start with having a look at the network. Perhaps I made a simple beginners misconfiguriation and thats the cause of all problems. How should we start @TheLumberjack ?

7

How about you send me a DM here on discourse with your network name and I'll have a look at it. Other than that, you could provision me an identity and service and I can also test from my machines, just to rule out your location as well if you want?

I'd recommend giving me one identity and making a service that the identity can both dial and bind. that's a very easy test... something like an http server is what I reach to since they are pretty easy to stand up on my own (docker or python etc makes it pretty easy)

That'd be what I'd say we should do for starters. These "bind timeout: no suitable edge router" errors are really unexpected.

8

I've been DM'ing with @MrRatherford... His CloudZiti had two WSS-eanabled edge routers. It appears (@tburtchell maybe can confirm) that a WSS enabled router cannot be used for "regular" (non-browzer) traffic.

I asked him to add a new non-WSS edge router. When it came online, the test service he setup for me stated working.

I think this comes down to a minor confusion around how many routers one needs in CloudZiti, but I think he should be all set now for both BrowZer and "regular" ziti traffic.

1 Like
9

Hi @MrRatherford, looks like your Edge Routers were updated and all connections you are making are now working? Thanks for helping get that straightened out @TheLumberjack ! As for the Home Assistant plugin, oddly enough there was a bug there too. The update I pushed through as 1.5.1 was a temporary remedy to address an issue with the CSDK tunnel that is built with the module (which occurs in versions 0.22.20 and 0.22.21 - thus staking at 0.22.19 to prevent it. The fix in the tunnel CSDK came through today and a new build was issued for the HA plugin (1.5.2). If you have auto update or initiate it, the version will update to latest. What that fix was is basically to address recursion for upstream resolution of DNS, which is depended upon in the plugin. That ensures you can resolve both Ziti private records and network/global records. the important bit of it is when ziti starts and after it has properly connected to the control because it would need to not loop to itself by asking the HA DNS container to resolve the controller.

Anyway - Should be fixed up alongside what you and @TheLumberjack talked about. Let us know if you have an issues with it, and a super big thanks for trying it out!

1 Like
10

Thanks for the detailed explanation. Wonderfull work you guys do here!

I updated the addon and it works as it should.

1 Like