# Ziti Router shutting down **URL:** \ **Category:** Support\ **Created:** [March 7, 2025, 8:33am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095 "2025-03-07T08:33:57Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 8:33am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/1 "2025-03-07T08:33:57Z")
Hi, Ziti Team! Could you please help me why my Ziti Router is shutting down? ```auto {"file":"github.com/openziti/ziti/ziti/router/run.go:145","func":"github.com/openziti/ziti/ziti/router.waitForShutdown","level":"info","msg":"shutting down ziti router","time":"2025-03-07T08:29:36.161Z"} {"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:42","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"error closing peer after rx error","time":"2025-03-07T08:29:36.161Z"} {"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"read tcp 10.0.64.135:33372-\u003e10.0.17.1:443: use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:44","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"rx error. closed peer and starting reconnection process","time":"2025-03-07T08:29:36.161Z"} {"file":"github.com/openziti/ziti/router/link/link_registry.go:307","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).Shutdown","level":"info","linkCount":0,"msg":"shutdown links in link registry","time":"2025-03-07T08:29:36.161Z"} {"_context":"tls:0.0.0.0:3022","error":"accept tcp [::]:3022: use of closed network connection","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:283","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"listener closed, exiting","time":"2025-03-07T08:29:36.161Z"} {"_context":"tls:0.0.0.0:3022","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:284","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"exited","time":"2025-03-07T08:29:36.161Z"} {"file":"github.com/openziti/ziti/tunnel/intercept/host/host.go:38","func":"github.com/openziti/ziti/tunnel/intercept/host.interceptor.Stop","level":"info","msg":"stopping host interceptor","time":"2025-03-07T08:29:36.161Z"} ``` ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 8:52am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/2 "2025-03-07T08:52:48Z")
I have openziti-controller and openziti-router running in a k8s cluster. This is the second iteration of my PoC, everything worked fine in the first one. I only changed the naming of the hosts and separated altIngress for the GUI login. I don't understand what the problem is, the router keeps crashing with an error, but it doesn't tell me anything. ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 8:54am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/3 "2025-03-07T08:54:28Z")
@TheLumberjack please help:) ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 9:10am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/4 "2025-03-07T09:10:31Z")
I remembered, I think I upgraded the helm chart version, I'll check it out ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 10:45am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/5 "2025-03-07T10:45:21Z")
@TheLumberjack I confirm that helm-chart 1.2.0 controller and 1.2.1 router versions are experiencing this error. I rolled back to 1.1.17 / 1.1.7 and everything works fine! Guys, obviously something where broken. Do I need to start an issue on GitHub? ```auto NAME CHART VERSION APP VERSION DESCRIPTION openziti/ziti-controller 1.2.1 1.4.2 Host an OpenZiti controller in Kubernetes openziti/ziti-controller 1.2.0 1.3.3 Host an OpenZiti controller in Kubernetes openziti/ziti-controller 1.1.17 1.2.2 Host an OpenZiti controller in Kubernetes openziti/ziti-router 1.2.2 1.4.2 Host an OpenZiti router in Kubernetes openziti/ziti-router 1.2.1 1.3.3 Host an OpenZiti router in Kubernetes openziti/ziti-router 1.2.0 1.3.3 Host an OpenZiti router in Kubernetes openziti/ziti-router 1.1.7 1.2.2 Host an OpenZiti router in Kubernetes ``` ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [March 7, 2025, 12:29pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/6 "2025-03-07T12:29:14Z")
Hi @kinseii, as a reminder our forum support is "as we can get to it" and we are primarily located at UTC-4/5. I check the forum multiple times throughout my day but everyone checks the forum as we can. If you need a greater level of support than that, there's always the NetFoundry option to manage and run the overlay network. 馃檪 As for what might be happening here, it's pretty difficult for me to have a clue as to what might be happening based on the 7 lines of logging. My expectation is that you have a misconfiguration in your network and it's causing the controller or router to stop because of a recent change I made to prevent starting up when the configuration is incorrect. Specifically this issue: [router: check advertised address on startup 路 Issue #2722 路 openziti/ziti 路 GitHub](https://github.com/openziti/ziti/issues/2722) I suspect you have an advertised address that is invalid for your configuration. You should inspect your configuration. Find all the `identity` blocks. They look like this: ```auto identity: cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-client.chain.pem" server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem" key: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/keys/ip-172-31-11-231-server.key" ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/cas.pem" alt_server_certs: - server_cert: "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/fullchain.pem" server_key: "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/privkey.pem" ``` From there, for every server\_cert in the config, verify the certificate is valid for the provided advertised address. For example in the above block, one of the two server\_certs shown **must** match this advertised address: ```auto ctrl: options: advertiseAddress: tls:ip-172-31-11-231:6262 ``` Using openssl, I can check it and I can see that the first one matches. ```auto openssl x509 -in /home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem -text | grep -A2 Alternative X509v3 Subject Alternative Name: DNS:ip-172-31-11-231, DNS:localhost, IP Address:127.0.0.1 ``` That would be my guess as to what's happening. This behavior begins with OpenZiti controller/router 1.4.x ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [March 7, 2025, 5:19pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/7 "2025-03-07T17:19:00Z")
If @TheLumberjack's hypothesis is correct, then you'll have a "fatal" message like this. ```bash kubectl logs --namespace miniziti --selector app.kubernetes.io/component=ziti-router --tail=-1 \ | grep '^\{' | jq 'select(.level|match("fatal|error"))' ``` ```json { "file": "github.com/openziti/ziti/router/config.go:918", "func": "github.com/openziti/ziti/router.LoadConfigWithOptions", "level": "fatal", "msg": "one or more advertise addresses are invalid: [invalid link.listeners.advertise: tls:miniziti-router.192.168.49.2.sslip.ion:443, error: identity is not valid for provided host: [miniziti-router.192.168.49.2.sslip.ion]. is valid for: [127.0.0.1, localhost, miniziti-router.192.168.49.2.sslip.io]]", "time": "2025-03-07T17:16:36.527Z" } ``` ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 11:53pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/8 "2025-03-07T23:53:33Z")
@TheLumberjack Yes, I realize that you can't answer quickly, I just tagged you because it seemed to me that you are one of those who is here often and always answers questions, at least my questions are always answered by you. I have to demonstrate the PoC to my management in a few days and can't do the tests yet. But I will try it in a week. Thanks! ---
**Author:** ![kinseii](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/kinseii/32/3140_2.png) [@kinseii](https://openziti.discourse.group/u/kinseii)\ **Post date:** [March 7, 2025, 11:55pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/9 "2025-03-07T23:55:37Z")
By the way, I have a question, I brought you only 7 lines of logs, because the rest is repeated, there was nothing new in these logs, and the question is, how can I enable debug via helm chart? Or can it be done through configmap option? ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [March 8, 2025, 12:58am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/10 "2025-03-08T00:58:30Z")
Yes, you can enable debug log level for ziti router or ziti controller by appending the `--verbose` run arg. In Kubernetes, this is done either temporarily by editing the deployment's pod template args like this: ```yaml spec: template: spec: containers: - args: - /etc/ziti/ziti-controller.yaml - --verbose ``` or permanently by setting a Helm chart input, e.g., ```bash --set image.additionalArgs=["--verbose"] ``` or ```yaml image: additionalArgs: - --verbose ``` EDIT: came back to clarify which one was temp vs. permanent ---
**Author:** ![fibbs](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/fibbs/32/3421_2.png) [@fibbs](https://openziti.discourse.group/u/fibbs)\ **Post date:** [April 20, 2025, 3:42pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/11 "2025-04-20T15:42:13Z")
Hey guys. I am having exactly the problem described by @TheLumberjack above. When trying to upgrade my controller, it complained that its certificate had no name for "ziti-controller-prometheus.svc.cluster.local" (or similar, sorry, don't have the exact error message anymore), which I temporarily fixed by deactivating the listener for prometheus metrics. Afterwards, I tried to upgrade a private router, and it complained about the same for the edge service. Both have in common that they are installed via the Helm Charts in Kubernetes Clusters and that I have let the Helm Charts and ziti components "do their thing" regarding certificates. It looks to me that the mechanism which is used to create certificates by default does actually put all the explicitly mentioned "advertised" names into the certificates, but not those that are "implicit", like the ".svc.cluster.local" names. How can I now change this in an existing installation? Can I somehow recreate those including the implicit names? Would I have to put those implicit names explicitly in the values.yamls? Had to downgrade my two private routers now, as I didn't get this solved for several hours myself. Thanks a lot in advance Christian ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [April 22, 2025, 3:34pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/12 "2025-04-22T15:34:31Z")
Welcome to the forum, @fibbs! You may have encountered [this issue](https://github.com/openziti/helm-charts/issues/315). A fix is being evaluated for the ziti-controller chart. ---
**Author:** ![fibbs](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/fibbs/32/3421_2.png) [@fibbs](https://openziti.discourse.group/u/fibbs)\ **Post date:** [April 22, 2025, 8:13pm UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/13 "2025-04-22T20:13:42Z")
Yes, it definitely is. And I am not surprised my good old friend @marvkis has discovered the same. Waiting for a fix then for both the controller and the router chart. ---
**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\ **Post date:** [April 23, 2025, 1:27am UTC](https://openziti.discourse.group/t/ziti-router-shutting-down/4095/14 "2025-04-23T01:27:18Z")
Here is the controller chart release with @marvkis fix for the prometheus scrape target's DNS SAN that satisfies the new startup checks that were introduced in Ziti 1.4.0: [Release ziti-controller-1.3.2 路 openziti/helm-charts 路 GitHub](https://github.com/openziti/helm-charts/releases/tag/ziti-controller-1.3.2) Next, let's diagnose your router problem. It, too, is failing Ziti 1.4.0+ startup check that requires a server cert with SAN for each advertised address in the config YAML, but that is where the similarities end. Please let me know which router chart version and Ziti version if not the default for the chart, and which configuration property specifies the advertised address that is missing.