Ziti-router: no network interface found for 0.0.0.0

Ziti Overlay
1

I would like to install a self-hosted instance of zrok. I follow the installation guide.
After I have started my ziti-router service and I see these errors.
It is unclear where I should look to fix the error.

{"addr":"tls:0.0.0.0:3022","error":"no network interface found for 0.0.0.0","file":"github.com/openziti/ziti/router/xlink_transport/config.go:76","func":"github.com/openziti/ziti/router/xlink_transport.loadListenerConfig","level":"warning","msg":"unable to get interface for address","time":"2025-04-15T15:13:01.179Z"}

{"action":"Create","entries":1,"error":"out of order event detected, currentIndex: 1, receivedIndex: 0, type :*common.ForgetfulEventCache","file":"github.com/openziti/ziti/common/router_data_model.go:444","func":"github.com/openziti/ziti/common.(*RouterDataModel).ApplyChangeSet","index":0,"level":"error","msg":"could not apply change set","synthetic":false,"time":"2025-04-15T15:13:01.254Z","type":"*edge_ctrl_pb.DataState_Event_PublicKey"}

{"action":"Create","entries":1,"error":"out of order event detected, currentIndex: 1, receivedIndex: 0, type :*common.ForgetfulEventCache","file":"github.com/openziti/ziti/common/router_data_model.go:444","func":"github.com/openziti/ziti/common.(*RouterDataModel).ApplyChangeSet","index":0,"level":"error","msg":"could not apply change set","synthetic":false,"time":"2025-04-15T15:13:01.255Z","type":"*edge_ctrl_pb.DataState_Event_PublicKey"}

/var/lib/private/ziti-router/config.yml:

link:
  dialers:
    - binding: transport
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:3022
      advertise:        tls:fqdn:3022
      options:
        outQueueSize:   4
1 Like
2

Can you bind anything to 0.0.0.0 at all? for example:

nc -l 0.0.0.0 12345

Is it possible you don't have IPv4 enabled? I would expect 0.0.0.0 to bind both ipv4 and ipv6. Do you need to use sudo for some reason?

3

Are we diagnosing this error warning?

{
  "addr": "tls:0.0.0.0:3022",
  "error": "no network interface found for 0.0.0.0",
  "file": "github.com/openziti/ziti/router/xlink_transport/config.go:76",
  "func": "github.com/openziti/ziti/router/xlink_transport.loadListenerConfig",
  "level": "warning",
  "msg": "unable to get interface for address",
  "time": "2025-04-15T15:13:01.179Z"
}

Is the symptom that ziti-router.service continually fails to start?

If the service did start, is it listening on port 3022? I expect not, but we can use this command later to verify that the problem was fixed.

❯ sudo lsof -Pnp $(systemctl show -p MainPID --value ziti-router.service ) |& grep 'TCP.*LISTEN'
ziti-router    622312 user    7u     IPv4            6797524      0t0      TCP *:3022 (LISTEN)

EDIT: I later realized the log level of this message is "warning," not "error"

4

No. It is here, regardless the error that I don't understand.

netstat -tnlp | grep 3022
tcp6       0      0 :::3022                 :::*                    LISTEN      533304/ziti
5

You are right. This is exactly this error. But the router seems to run. Might be I can continue with the installation of zrok.

6

I think it's safe to ignore this warning-level message if it's functioning as expected.

  "level": "warning",
7

thank you,

what happens with the network if for some reason the host ZROK API ENDPOINT becomes unavailable, unfortunately.

Should I deploy two controllers connected to the same postgres database? Probably the network will be dead.

The second question concerns the access side:
Does zrok access private need ZROK API ENDPOINT to function after the access has been created? The access side needs only edge routers, isn't it?

Well I see that it is used :frowning:

$systemctl --user restart zrok-agent.service

$ for i in  $(dig api-v1.zrok.io +short); do netstat -tn | grep $i; done
tcp        0      0 192.168.7.12:42716      34.200.133.147:443      ESTABLISHED
tcp        0      0 192.168.7.12:42714      34.200.133.147:443      ESTABLISHED
8

You'll be unable to manage shares and accesses or access the web console if the zrok controller isn't available.

An array of zrok controllers will behave as a single zrok instance if they share a PostgreSQL data source, which provides table locking.

You may scale out the zrok public frontends at will, optionally cloning a shared public frontend zrok environment since they always have the same authorizations.

The oauth frontend scales with the public frontend.

This PostgreSQL-specific locking configuration may be relevant if you're enforcing limits: Configuring Limits | zrok

9

This is great. Where I can read about it. It is doable to install two on two hosts.

Do you mean that I can repeat the installation of a controller on another host pointing to the same postgreSQL database in etc/ctrl.yml?

10

That's how I expect it to work. You may be the first to attempt it. I assume you will need a load balancer or DNS round robin for each array of controllers and frontends.

You will probably have only one, but each public frontend must scale independently behind a LB (or RR) where a wildcard DNS record resolves and wildcard certificate is bound, corresponding to that frontend's public share URL template.

11

Starting the second zrok controller was easy. But another problem appears: the zrok-agent needs also the ziti controller to function.
The first host runs the zit controller, 6 ziti routers and the first zrok conroller.
The second host runs only the second zrok controller and a ziti router.
So if the first host is unavailable the network stops to function because the ziti controller is unreachable.

I have not configured HA. I don't know whether the zrok-agent will function with HA ziti controllers.
Apparently if the ziti controller is not available there are no connections to routers.

12

Correct, the ziti controller is required frequently, but not continually, for zrok shares and accesses to function. The ziti controller provides ziti service and router discovery for the ziti identity used by the zrok agent, so the agent will function with the services (shares) and routers it has already discovered until the session expires if the ziti controller is unavailable.

I think ziti controller clustering is needed to remove all single points of failure, but as we learned in another topic, zrok has not yet implemented ziti HA, so you are truly blazing a new trail here!

13

Can we expect to see an implementation for HA controllers in 2025?

14

That seems very likely.

15

This is great!
Thank you very much.

Let me another question. I have 2 services sock5 and element, they are both on the same machine. And I have 2 zrok controllers on ovh29 and dc67. I have noticed that sock5 service picks up the right terminator (ovh29), but element picks up the wrong one dc67 (miles away from the the host)

Now I have the following circuits for element's flow: ovh76-> dc67 instead of ovh76-> ovh29. Well because the terminator for the service is wrong.
It is certainly because I made a mistake by setting ZROK_API_ENDPOINT="https://public.domain.name:port". And dns has 2 addresses for this name, precisely the addresses of ovh29 and dc67 where zrok controllers are.
Is there a way to change the terminator? Or I need to release the share and recreate it. It is not clear how to specify the terminator's router. Exemple:

zrok share private -b drive path

╭───────────┬───────────────────────────┬─────────┬────────────────────────┬─────────────────────┬─────────────────────────────────────────────────╮
│ ID        │ CLIENT                    │ SERVICE │ TERMINATOR             │ CREATEDAT           │ PATH                                            │
├───────────┼───────────────────────────┼─────────┼────────────────────────┼─────────────────────┼─────────────────────────────────────────────────┤
│ Jwn9pedoc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 17:45:00 │ r/ovh76 -> l/5PwHQ3EQzxzhuxq05rGnFx -> r/ovh29  │
│ TdGlneJoc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 17:49:15 │ r/ovh76 -> l/5PwHQ3EQzxzhuxq05rGnFx -> r/ovh29  │
│ VuH9muJRc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 16:35:26 │ r/ovh221 -> l/2eRaE0iG0diFMDPrX4TJbZ -> r/ovh29 │
│ ZUzpuedoc │ cma2gcvgp10okr1j4t5avxtns │ element │ 5jWewbMLAy3IyC1T2HPOUB │ 2025-04-29 17:51:40 │ r/ovh76 -> l/61i116afGfeEls9CcUxGoz -> r/dc67   │
│ ftjAnedRc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 17:50:12 │ r/ovh76 -> l/5PwHQ3EQzxzhuxq05rGnFx -> r/ovh29  │
│ hrXdsCdRc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 16:32:58 │ r/ovh29                                         │
│ ilkEpwdoc │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 17:41:03 │ r/ovh76 -> l/5PwHQ3EQzxzhuxq05rGnFx -> r/ovh29  │
│ rkznCwdRc │ cma2gcvgp10okr1j4t5avxtns │ element │ 5jWewbMLAy3IyC1T2HPOUB │ 2025-04-29 17:51:40 │ r/ovh76 -> l/61i116afGfeEls9CcUxGoz -> r/dc67   │
│ wW-VmuJo5 │ cma2gcwkr10opr1j4k3tpe8b3 │ socks5  │ 6Uz4d4tkjy6PAlbkPIOeTY │ 2025-04-29 16:38:31 │ r/ovh76 -> l/5PwHQ3EQzxzhuxq05rGnFx -> r/ovh29  │
╰───────────┴───────────────────────────┴─────────┴────────────────────────┴─────────────────────┴─────────────────────────────────────────────────╯


╭────────────────────────┬─────────┬────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────┬────────────╮
│ ID                     │ SERVICE │ ROUTER │ BINDING │ ADDRESS                │ INSTANCE │ COST │ PRECEDENCE │ DYNAMIC COST │ HOST ID    │
├────────────────────────┼─────────┼────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┼────────────┤
│ 5jWewbMLAy3IyC1T2HPOUB │ element │ dc67   │ edge    │ 5jWewbMLAy3IyC1T2HPOUB │          │    0 │ default    │            2 │ qNTXfYbyhW │
│ 6Uz4d4tkjy6PAlbkPIOeTY │ socks5  │ ovh29  │ edge    │ 6Uz4d4tkjy6PAlbkPIOeTY │          │    0 │ default    │           26 │ qNTXfYbyhW │
╰────────────────────────┴─────────┴────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────┴────────────╯


╭────────────────────────┬────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                     │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├────────────────────────┼────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 16E2TczYESHKahbg3m38uo │ dc67   │ ovh223   │           1 │      17.7ms │      18.1ms │ Connected │     up │        36 │
│ 1AAwqehT0saVbe5iVa4xiA │ dc67   │ ovh29    │           1 │      16.5ms │      16.4ms │ Connected │     up │        33 │
│ 1AlTJZCRfaKmwiyV7EEjCh │ ovh76  │ ovh221   │           1 │       2.8ms │       2.7ms │ Connected │     up │         5 │
│ 1Cj1nteTAOuVeaTgtcuhBU │ dc67   │ ovh3     │           1 │      19.4ms │      19.3ms │ Connected │     up │        39 │
│ 1DRZUk4wHiWL6u5nSOdMq5 │ ovh3   │ ovh89    │           1 │       2.8ms │       2.8ms │ Connected │     up │         5 │
│ 1blqRd6d3puyPr1zhljc2y │ ovh3   │ ovh221   │           1 │       2.8ms │       2.7ms │ Connected │     up │         5 │
│ 1gcphw7HfD6GnrR1FtC3Sp │ ovh223 │ dc67     │           1 │      19.2ms │      18.8ms │ Connected │     up │        38 │
│ 1kUKIHJuQGbdZQyOEP6WmI │ ovh76  │ ovh223   │           1 │       2.8ms │       2.8ms │ Connected │     up │         5 │
│ 1mk00fyR7GmeiAaHeEytdY │ ovh76  │ ovh3     │           1 │       2.7ms │       2.7ms │ Connected │     up │         5 │
│ 1sKEAKcGxBQUc8Fq751uJM │ ovh76  │ ovh89    │           1 │       2.9ms │       2.7ms │ Connected │     up │         5 │
╰────────────────────────┴────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯

Why the COST of edge-routers is always 0. I guess all connection related issues should be included in the edge router's COST. Since the link's COST is different, it mostly depends on the distance between the routers.

╭────────────┬────────┬────────┬───────────────┬──────┬────────────╮
│ ID         │ NAME   │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼────────┼────────┼───────────────┼──────┼────────────┤
│ 7H8BaaKm9F │ ovh29  │ true   │ true          │    0 │            │
│ JI-BBYbyIW │ ovh76  │ true   │ true          │    0 │            │
│ JoOo6WCyhW │ ovh221 │ true   │ true          │    0 │            │
│ LlqloYCyhW │ ovh3   │ true   │ true          │    0 │            │
│ T8.KBWbyhW │ ovh89  │ true   │ true          │    0 │            │
│ WIZxdWbqhW │ dc67   │ true   │ true          │    0 │            │
│ Y-UUoWCyIW │ ovh223 │ true   │ true          │    0 │            │
╰────────────┴────────┴────────┴───────────────┴──────┴────────────╯

After restarting I see that element has wrong router again (dc67). It is unclear why ziti allocates a terminator on the remote router.

╭────────────────────────┬─────────┬────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────┬────────────╮
│ ID                     │ SERVICE │ ROUTER │ BINDING │ ADDRESS                │ INSTANCE │ COST │ PRECEDENCE │ DYNAMIC COST │ HOST ID    │
├────────────────────────┼─────────┼────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┼────────────┤
│ 2mdoEbtVprBVKYJQKlOtYR │ element │ dc67   │ edge    │ 2mdoEbtVprBVKYJQKlOtYR │          │    0 │ default    │            4 │ oOesPw1m9F │
│ 6TJfsrSiFqEp6HNdfovgEw │ socks5  │ ovh76  │ edge    │ 6TJfsrSiFqEp6HNdfovgEw │          │    0 │ default    │           38 │ oOesPw1m9F │
╰────────────────────────┴─────────┴────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────┴────────────╯


╭────────────┬───────────────────────────┬─────────┬────────────────────────┬─────────────────────┬─────────────────────────────────────────────────╮
│ ID         │ CLIENT                    │ SERVICE │ TERMINATOR             │ CREATEDAT           │ PATH                                            │
├────────────┼───────────────────────────┼─────────┼────────────────────────┼─────────────────────┼─────────────────────────────────────────────────┤
│ 2f91yi.5e  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:33:05 │ r/ovh223 -> l/35Mlsz88K9xtbVFdZrjprO -> r/ovh76 │
│ 3xd-yC.EF  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:36:43 │ r/ovh3 -> l/5dZ79Lgsiy57Aa7jAfjAx7 -> r/ovh76   │
│ 3xqSlC.5F  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:36:56 │ r/ovh3 -> l/5dZ79Lgsiy57Aa7jAfjAx7 -> r/ovh76   │
│ 4MRBli.5e  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:37:59 │ r/ovh3 -> l/5dZ79Lgsiy57Aa7jAfjAx7 -> r/ovh76   │
│ 9v-QlCsEe  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:30:32 │ r/dc67 -> l/7b6PH9TQqHwz9UpCm9ZCN2 -> r/ovh76   │
│ AWtWyC.5eG │ cma350qxz00vi4tj4elqvn4x5 │ element │ 2mdoEbtVprBVKYJQKlOtYR │ 2025-04-29 23:38:29 │ r/ovh29 -> l/4spD4qO6o27PlEz9bvkM1n -> r/dc67   │
│ BaLSyi.5F  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:36:47 │ r/ovh3 -> l/5dZ79Lgsiy57Aa7jAfjAx7 -> r/ovh76   │
│ D4NLlCsEF  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:27:46 │ r/dc67 -> l/7b6PH9TQqHwz9UpCm9ZCN2 -> r/ovh76   │
│ DwOSyCs5e  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:36:49 │ r/ovh3 -> l/5dZ79Lgsiy57Aa7jAfjAx7 -> r/ovh76   │
│ ELWLlis5F  │ cma350rjq00vl4tj4fctlp7mt │ socks5  │ 6TJfsrSiFqEp6HNdfovgEw │ 2025-04-29 23:27:51 │ r/dc67 -> l/7b6PH9TQqHwz9UpCm9ZCN2 -> r/ovh76   │
╰────────────┴───────────────────────────┴─────────┴────────────────────────┴─────────────────────┴─────────────────────────────────────────────────╯

So I need to restart the router dc67 to get a correct circuits

╭────────────────────────┬─────────┬────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────┬────────────╮
│ ID                     │ SERVICE │ ROUTER │ BINDING │ ADDRESS                │ INSTANCE │ COST │ PRECEDENCE │ DYNAMIC COST │ HOST ID    │
├────────────────────────┼─────────┼────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┼────────────┤
│ 2fnPDU1qqTPhryBlrczwg1 │ element │ ovh223 │ edge    │ 2fnPDU1qqTPhryBlrczwg1 │          │    0 │ default    │            2 │ oOesPw1m9F │
│ 6TJfsrSiFqEp6HNdfovgEw │ socks5  │ ovh76  │ edge    │ 6TJfsrSiFqEp6HNdfovgEw │          │    0 │ default    │            8 │ oOesPw1m9F │
╰────────────────────────┴─────────┴────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────┴────────────╯

This is strange: why the terminator was initially allocated on the remote host? So I need to monitor this manually and restart the wrong router to initiate the rerouting.

16

ZROK_API_ENDPOINT defines the URL of the zrok controller API only, so it will not influence the Ziti terminators that are created by the hosting identity on the best available router at the time the service hosting begins.

I'm not so sure about terminator selection or link cost. Generally, I expect those to be autonomous unless I need to adjust the static cost of a router so it's used more often or less often. It might help someone else to analyze your findings if you precede each output with a heading like "terminators", "circuits", "links", etc. Good luck.