Ziti Router shutting down

Support
1

Hi, Ziti Team!
Could you please help me why my Ziti Router is shutting down?

{"file":"github.com/openziti/ziti/ziti/router/run.go:145","func":"github.com/openziti/ziti/ziti/router.waitForShutdown","level":"info","msg":"shutting down ziti router","time":"2025-03-07T08:29:36.161Z"}
{"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:42","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"error closing peer after rx error","time":"2025-03-07T08:29:36.161Z"}
{"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"read tcp 10.0.64.135:33372-\u003e10.0.17.1:443: use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:44","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"rx error. closed peer and starting reconnection process","time":"2025-03-07T08:29:36.161Z"}
{"file":"github.com/openziti/ziti/router/link/link_registry.go:307","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).Shutdown","level":"info","linkCount":0,"msg":"shutdown links in link registry","time":"2025-03-07T08:29:36.161Z"}
{"_context":"tls:0.0.0.0:3022","error":"accept tcp [::]:3022: use of closed network connection","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:283","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"listener closed, exiting","time":"2025-03-07T08:29:36.161Z"}
{"_context":"tls:0.0.0.0:3022","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:284","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"exited","time":"2025-03-07T08:29:36.161Z"}
{"file":"github.com/openziti/ziti/tunnel/intercept/host/host.go:38","func":"github.com/openziti/ziti/tunnel/intercept/host.interceptor.Stop","level":"info","msg":"stopping host interceptor","time":"2025-03-07T08:29:36.161Z"}
2

I have openziti-controller and openziti-router running in a k8s cluster. This is the second iteration of my PoC, everything worked fine in the first one. I only changed the naming of the hosts and separated altIngress for the GUI login. I don't understand what the problem is, the router keeps crashing with an error, but it doesn't tell me anything.

3

@TheLumberjack please help:)

4

I remembered, I think I upgraded the helm chart version, I'll check it out

5

@TheLumberjack
I confirm that helm-chart 1.2.0 controller and 1.2.1 router versions are experiencing this error. I rolled back to 1.1.17 / 1.1.7 and everything works fine! Guys, obviously something where broken. Do I need to start an issue on GitHub?

NAME                            CHART VERSION   APP VERSION     DESCRIPTION
openziti/ziti-controller        1.2.1           1.4.2           Host an OpenZiti controller in Kubernetes
openziti/ziti-controller        1.2.0           1.3.3           Host an OpenZiti controller in Kubernetes
openziti/ziti-controller        1.1.17          1.2.2           Host an OpenZiti controller in Kubernetes

openziti/ziti-router    1.2.2           1.4.2           Host an OpenZiti router in Kubernetes
openziti/ziti-router    1.2.1           1.3.3           Host an OpenZiti router in Kubernetes
openziti/ziti-router    1.2.0           1.3.3           Host an OpenZiti router in Kubernetes
openziti/ziti-router    1.1.7           1.2.2           Host an OpenZiti router in Kubernetes
6

Hi @kinseii, as a reminder our forum support is "as we can get to it" and we are primarily located at UTC-4/5. I check the forum multiple times throughout my day but everyone checks the forum as we can. If you need a greater level of support than that, there's always the NetFoundry option to manage and run the overlay network. :slight_smile:

As for what might be happening here, it's pretty difficult for me to have a clue as to what might be happening based on the 7 lines of logging. My expectation is that you have a misconfiguration in your network and it's causing the controller or router to stop because of a recent change I made to prevent starting up when the configuration is incorrect. Specifically this issue: router: check advertised address on startup · Issue #2722 · openziti/ziti · GitHub

I suspect you have an advertised address that is invalid for your configuration. You should inspect your configuration. Find all the identity blocks. They look like this:

identity:
  cert:        "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-client.chain.pem"
  server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem"
  key:         "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/keys/ip-172-31-11-231-server.key"
  ca:          "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/cas.pem"
  alt_server_certs:
    - server_cert:  "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/fullchain.pem"
      server_key:   "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/privkey.pem"

From there, for every server_cert in the config, verify the certificate is valid for the provided advertised address. For example in the above block, one of the two server_certs shown must match this advertised address:

ctrl:
  options:
    advertiseAddress: tls:ip-172-31-11-231:6262

Using openssl, I can check it and I can see that the first one matches.

openssl x509 -in /home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem -text | grep -A2 Alternative
            X509v3 Subject Alternative Name:
                DNS:ip-172-31-11-231, DNS:localhost, IP Address:127.0.0.1

That would be my guess as to what's happening. This behavior begins with OpenZiti controller/router 1.4.x

7

If @TheLumberjack's hypothesis is correct, then you'll have a "fatal" message like this.

kubectl logs --namespace miniziti --selector app.kubernetes.io/component=ziti-router --tail=-1 \
| grep '^\{' | jq 'select(.level|match("fatal|error"))'

{
  "file": "github.com/openziti/ziti/router/config.go:918",
  "func": "github.com/openziti/ziti/router.LoadConfigWithOptions",
  "level": "fatal",
  "msg": "one or more advertise addresses are invalid: [invalid link.listeners.advertise: tls:miniziti-router.192.168.49.2.sslip.ion:443, error: identity is not valid for provided host: [miniziti-router.192.168.49.2.sslip.ion]. is valid for: [127.0.0.1, localhost, miniziti-router.192.168.49.2.sslip.io]]",
  "time": "2025-03-07T17:16:36.527Z"
}
8

@TheLumberjack
Yes, I realize that you can't answer quickly, I just tagged you because it seemed to me that you are one of those who is here often and always answers questions, at least my questions are always answered by you.

I have to demonstrate the PoC to my management in a few days and can't do the tests yet. But I will try it in a week. Thanks!

9

By the way, I have a question, I brought you only 7 lines of logs, because the rest is repeated, there was nothing new in these logs, and the question is, how can I enable debug via helm chart? Or can it be done through configmap option?

10

Yes, you can enable debug log level for ziti router or ziti controller by appending the --verbose run arg.

In Kubernetes, this is done either temporarily by editing the deployment's pod template args like this:

spec:                                                                                                                                                                                                                                                                                                                                                                                                                                       
  template:                                                                                                                                                                                                                                                                                                                                                                                                                                 
    spec:                                                                                                                                                                                                                                                                                                                                                                                                                                   
      containers:                                                                                                                                                                                                                                                                                                                                                                                                                           
      - args:                                                                                                                                                                                                                                                                                                                                                                                                                               
        - /etc/ziti/ziti-controller.yaml
        - --verbose

or permanently by setting a Helm chart input, e.g.,

--set image.additionalArgs=["--verbose"]

or

image:
  additionalArgs:
    - --verbose

EDIT: came back to clarify which one was temp vs. permanent

1 Like
11

Hey guys. I am having exactly the problem described by @TheLumberjack above. When trying to upgrade my controller, it complained that its certificate had no name for "ziti-controller-prometheus.svc.cluster.local" (or similar, sorry, don't have the exact error message anymore), which I temporarily fixed by deactivating the listener for prometheus metrics. Afterwards, I tried to upgrade a private router, and it complained about the same for the edge service. Both have in common that they are installed via the Helm Charts in Kubernetes Clusters and that I have let the Helm Charts and ziti components "do their thing" regarding certificates. It looks to me that the mechanism which is used to create certificates by default does actually put all the explicitly mentioned "advertised" names into the certificates, but not those that are "implicit", like the ".svc.cluster.local" names. How can I now change this in an existing installation? Can I somehow recreate those including the implicit names? Would I have to put those implicit names explicitly in the values.yamls?

Had to downgrade my two private routers now, as I didn't get this solved for several hours myself.

Thanks a lot in advance

Christian

12

Welcome to the forum, @fibbs!

You may have encountered this issue. A fix is being evaluated for the ziti-controller chart.

13

Yes, it definitely is. And I am not surprised my good old friend @marvkis has discovered the same. Waiting for a fix then for both the controller and the router chart.

1 Like
14

Here is the controller chart release with @marvkis fix for the prometheus scrape target's DNS SAN that satisfies the new startup checks that were introduced in Ziti 1.4.0: Release ziti-controller-1.3.2 · openziti/helm-charts · GitHub

Next, let's diagnose your router problem. It, too, is failing Ziti 1.4.0+ startup check that requires a server cert with SAN for each advertised address in the config YAML, but that is where the similarities end.

Please let me know which router chart version and Ziti version if not the default for the chart, and which configuration property specifies the advertised address that is missing.