Graceful shutdown of 'ziti tunnel proxy'

Hi there,
we are running a ziti proxy on one of our machines like:

/usr/local/bin/ziti tunnel proxy Service_TEST:3000  -i /home/<usr>/.config/ziti/identity.json

As we want to shutdown this ziti proxy gracefully, to avoid loose hanging sessions in the openziti controller, we've tried sending linux signals to the ziti process e.g.

kill -SIGTERM <pid>

Unfortunately it seems that the ziti process does not react to SIGTERM or SIGINT, we can only kill it via SIGKILL.

Are there any better ways of shutting down the ziti proxy process gracefully ?

BR
Jan

This seems to be an issue that's occured recently. There were two bugs filled that are in this area. Did you happen to stop the controller first?

See: ziti router never exits if it can't connect to the controller · Issue #1755 · openziti/ziti · GitHub and router run - ignores TERM and QUIT signals when unhappy · Issue #1784 · openziti/ziti · GitHub

The expectation that SIGINT should terminate the process correctly. You should not need to kill. Do you have and extra information about steps to reproduce your problem? Can you reliably trigger the issue? It happened for me when the router couldn't contact the controller only and worked fine most of the time, that's why I'm wondering if your flow might be triggering the issue? You might have some steps to reproduce that are different.

Thanks

Hi @TheLumberjack,
thanks for responding :slight_smile:

Did you happen to stop the controller first?

No our approach is stopping the ziti process independent from the openziti controller.

Issues linked:

As I understand this is about a ziti router, in our case this is just running the ziti binary in proxy mode - or do I get sth. wrong here ? :slight_smile:
We are doing this as we just want to forward a dedicated port from a local working evironment...

The expectation that SIGINT should terminate the process correctly. You should not need to kill. Do you have and extra information about steps to reproduce your problem? Can you reliably trigger the issue?

In my/our case the ziti process does not react to SIGINT signal consistently. So yes triggering this issue can be reproduced successfully every time - on our side at least.

ziti version is v0.28.4

-> Just tried with ziti v0.32.2, here it seems to work but gives me some errors:

[  11.188]    INFO ziti/tunnel/intercept/proxy.(*interceptor).Stop: stopping proxy interceptor
[  11.188]   ERROR agent.(*handler).listen: {error=[accept unix /tmp/gops-agent.2380.sock: use of closed network connection]} error accepting gops connection, closing gops listener
[  11.188]   ERROR agent.(*handler).listen.func1: {error=[close unix /tmp/gops-agent.2380.sock: use of closed network connection]} error closing gops listener

How was my workflow ? Just executing locally on my machine:

./ziti tunnel proxy Service_TEST:3000  --identity-dir /opt/openziti/etc/identities/

Then getting the PID via netstat -tulpn and signaling via kill -SIGINT <PID>

Oh! I didn't realize you were using the ziti tunnel. My apologies, it just sounded so similar my brain just associated them both.

So you're just starting Ziti tunnel and then trying to terminate it, yeah that should behave. I'll try to reproduce this issue myself. Thanks for clarifying. (ziti router and ziti tunnel are both in the same ziti binary. My guess is the same issue affects them both)

I was able to reproduce the problem using 0.28.4 of the ziti CLI but not with the latest version as you also noted. Seems like the bug has been fixed for ziti tunnel in subsequent releases since 0.28.4.

The GOPS errors shouldn't be important but I'll track those down and see if we can suppress those as well, but upgrading to 0.32.2 seems like the best approach to be able to stop the ziti tunnel without killing the process.

The ziti-edge-tunnel will support proxying soon. When it does, I would recommend changing over to using ziti-edge-tunnel. As you probably know, ziti tunnel has been deprecated and I believe it's likely to be removed some day.

1 Like