OpenZiti router stuck at 100% CPU

Ziti Overlay
1

I'm seeing an issue where one of my OpenZiti routers periodically gets stuck using 100% of one CPU core.

I'm running OpenZiti 2.0.6.

I caught it while it was happening and took a 30-second CPU pprof. Almost all of the CPU time was here:

github.com/openziti/ziti/v2/common/metrics.(*controllersReporter).AcceptMetrics
github.com/openziti/metrics.(*usageRegistryImpl).sendMsgs

The Go stack at the same time showed:

goroutine 114 [runnable]:

github.com/openziti/ziti/v2/common/metrics.(*controllersReporter).AcceptMetrics
    common/metrics/ctrl_reporter.go:36

github.com/openziti/metrics.(*usageRegistryImpl).sendMsgs
    usage_registry.go:292

The router logs also have messages like:

no controllers available to notify of link

and:

no controller available, cannot create circuit

I took a look at common/metrics/ctrl_reporter.go, and I think there may be a tight loop when there are queued metrics but no controllers available.

The code loops while reporter.msgs isn't empty, but if reporter.ctrls.GetAll() returns no controllers, nothing removes the first message or exits the outer loop.

That seems to match what I'm seeing: once the router gets into this state, one thread stays at basically 100% CPU.

I checked the 2.0.7 source as well and it looks like this part of AcceptMetrics() is still the same.

I have the full CPU pprof, Go stack dump, router logs, and diagnostics from while it was happening if those would be useful.

2

I'm sure @plorenz will be interested. Thanks for capturing that. You can send DMs here on discourse if you want else we can figure out some way to get them to us.

3

I DM'd @plorenz with a diag log package.

4

Thank you for reporting the issue (and digging in). I'm taking a look.

Paul

5

Fix plus backports are up. The 2.0 backport is here: [Backport-2.0] Stop the router spinning on metrics while no controller is registered by plorenz · Pull Request #4507 · openziti/ziti · GitHub and should be in the 2.0.8 release.

Thanks again for reporting!
Paul

6

Thank you for the attention and super fast fix! I'll wait until the 2.0.8 GA release to test.