Hi, Ziti Team!
Could you please help me why my Ziti Router is shutting down?
{"file":"github.com/openziti/ziti/ziti/router/run.go:145","func":"github.com/openziti/ziti/ziti/router.waitForShutdown","level":"info","msg":"shutting down ziti router","time":"2025-03-07T08:29:36.161Z"}
{"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:42","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"error closing peer after rx error","time":"2025-03-07T08:29:36.161Z"}
{"_context":"u{reconnecting}-\u003ei{ziti-controller-ctrl-plane-client-identity/65Y1}","error":"read tcp 10.0.64.135:33372-\u003e10.0.17.1:443: use of closed network connection","file":"github.com/openziti/channel/v3@v3.0.37/reconnecting_impl.go:44","func":"github.com/openziti/channel/v3.(*reconnectingImpl).Rx","level":"error","msg":"rx error. closed peer and starting reconnection process","time":"2025-03-07T08:29:36.161Z"}
{"file":"github.com/openziti/ziti/router/link/link_registry.go:307","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).Shutdown","level":"info","linkCount":0,"msg":"shutdown links in link registry","time":"2025-03-07T08:29:36.161Z"}
{"_context":"tls:0.0.0.0:3022","error":"accept tcp [::]:3022: use of closed network connection","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:283","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"listener closed, exiting","time":"2025-03-07T08:29:36.161Z"}
{"_context":"tls:0.0.0.0:3022","file":"github.com/openziti/transport/v2@v2.0.165/tls/listener.go:284","func":"github.com/openziti/transport/v2/tls.(*sharedListener).runAccept","level":"info","msg":"exited","time":"2025-03-07T08:29:36.161Z"}
{"file":"github.com/openziti/ziti/tunnel/intercept/host/host.go:38","func":"github.com/openziti/ziti/tunnel/intercept/host.interceptor.Stop","level":"info","msg":"stopping host interceptor","time":"2025-03-07T08:29:36.161Z"}
I have openziti-controller and openziti-router running in a k8s cluster. This is the second iteration of my PoC, everything worked fine in the first one. I only changed the naming of the hosts and separated altIngress for the GUI login. I don't understand what the problem is, the router keeps crashing with an error, but it doesn't tell me anything.
@TheLumberjack please help:)
I remembered, I think I upgraded the helm chart version, I'll check it out
@TheLumberjack
I confirm that helm-chart 1.2.0 controller and 1.2.1 router versions are experiencing this error. I rolled back to 1.1.17 / 1.1.7 and everything works fine! Guys, obviously something where broken. Do I need to start an issue on GitHub?
NAME CHART VERSION APP VERSION DESCRIPTION
openziti/ziti-controller 1.2.1 1.4.2 Host an OpenZiti controller in Kubernetes
openziti/ziti-controller 1.2.0 1.3.3 Host an OpenZiti controller in Kubernetes
openziti/ziti-controller 1.1.17 1.2.2 Host an OpenZiti controller in Kubernetes
openziti/ziti-router 1.2.2 1.4.2 Host an OpenZiti router in Kubernetes
openziti/ziti-router 1.2.1 1.3.3 Host an OpenZiti router in Kubernetes
openziti/ziti-router 1.2.0 1.3.3 Host an OpenZiti router in Kubernetes
openziti/ziti-router 1.1.7 1.2.2 Host an OpenZiti router in Kubernetes
Hi @kinseii, as a reminder our forum support is "as we can get to it" and we are primarily located at UTC-4/5. I check the forum multiple times throughout my day but everyone checks the forum as we can. If you need a greater level of support than that, there's always the NetFoundry option to manage and run the overlay network. 
As for what might be happening here, it's pretty difficult for me to have a clue as to what might be happening based on the 7 lines of logging. My expectation is that you have a misconfiguration in your network and it's causing the controller or router to stop because of a recent change I made to prevent starting up when the configuration is incorrect. Specifically this issue: router: check advertised address on startup · Issue #2722 · openziti/ziti · GitHub
I suspect you have an advertised address that is invalid for your configuration. You should inspect your configuration. Find all the identity blocks. They look like this:
identity:
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-client.chain.pem"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/keys/ip-172-31-11-231-server.key"
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/cas.pem"
alt_server_certs:
- server_cert: "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/fullchain.pem"
server_key: "/data/docker/letsencrypt/live/zrok.clint.demo.openziti.org/privkey.pem"
From there, for every server_cert in the config, verify the certificate is valid for the provided advertised address. For example in the above block, one of the two server_certs shown must match this advertised address:
ctrl:
options:
advertiseAddress: tls:ip-172-31-11-231:6262
Using openssl, I can check it and I can see that the first one matches.
openssl x509 -in /home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-intermediate/certs/ip-172-31-11-231-server.chain.pem -text | grep -A2 Alternative
X509v3 Subject Alternative Name:
DNS:ip-172-31-11-231, DNS:localhost, IP Address:127.0.0.1
That would be my guess as to what's happening. This behavior begins with OpenZiti controller/router 1.4.x
If @TheLumberjack's hypothesis is correct, then you'll have a "fatal" message like this.
kubectl logs --namespace miniziti --selector app.kubernetes.io/component=ziti-router --tail=-1 \
| grep '^\{' | jq 'select(.level|match("fatal|error"))'
{
"file": "github.com/openziti/ziti/router/config.go:918",
"func": "github.com/openziti/ziti/router.LoadConfigWithOptions",
"level": "fatal",
"msg": "one or more advertise addresses are invalid: [invalid link.listeners.advertise: tls:miniziti-router.192.168.49.2.sslip.ion:443, error: identity is not valid for provided host: [miniziti-router.192.168.49.2.sslip.ion]. is valid for: [127.0.0.1, localhost, miniziti-router.192.168.49.2.sslip.io]]",
"time": "2025-03-07T17:16:36.527Z"
}
@TheLumberjack
Yes, I realize that you can't answer quickly, I just tagged you because it seemed to me that you are one of those who is here often and always answers questions, at least my questions are always answered by you.
I have to demonstrate the PoC to my management in a few days and can't do the tests yet. But I will try it in a week. Thanks!
By the way, I have a question, I brought you only 7 lines of logs, because the rest is repeated, there was nothing new in these logs, and the question is, how can I enable debug via helm chart? Or can it be done through configmap option?
Yes, you can enable debug log level for ziti router or ziti controller by appending the --verbose run arg.
In Kubernetes, this is done either temporarily by editing the deployment's pod template args like this:
spec:
template:
spec:
containers:
- args:
- /etc/ziti/ziti-controller.yaml
- --verbose
or permanently by setting a Helm chart input, e.g.,
--set image.additionalArgs=["--verbose"]
or
image:
additionalArgs:
- --verbose
EDIT: came back to clarify which one was temp vs. permanent
1 Like