I am doing some debugging on my gloang reverse proxy.. that could be creating problems with the ziti controller.
when I run the command
ziti ps stack
I receive the following error
error: too many gops-agent process found, including [ziti-controller (pid 3352565), ziti-router (pid 3355101)]
Any tips on what this means or how to fix?
I found a tool called gops..
This is what I found when I run it.
860003 1 agent unknown Go version /usr/libexec/oracle-cloud-agent/agent
860056 1 updater unknown Go version /usr/libexec/oracle-cloud-agent/updater/updater
860072 860003 gomon unknown Go version /usr/libexec/oracle-cloud-agent/plugins/gomon/gomon
860177 860164 runcommand unknown Go version /usr/libexec/oracle-cloud-agent/plugins/runcommand/runcommand
3362128 3361365 gops go1.17.2 /root/go/bin/gops
3348505 3348364 rproxy go1.17.2 /home/opc/.ziti/quickstart/instance-20220416-1603/sdk-golang/exercises/http/server/rproxy/rproxy
3352565 1 ziti-controller go1.17.8 /home/opc/.ziti/quickstart/instance-20220416-1603/ziti-bin/ziti-v0.25.4/ziti-controller
3355101 3349256 ziti-router go1.17.8 /home/opc/.ziti/quickstart/instance-20220416-1603/ziti-bin/ziti-v0.25.4/ziti-router
Does this mean that there are too many go apps running on my server?
What I will try is to move the reverse proxy to another server to see if this helps
Let me know if you have any other tips..
What I did was to stop the router… this appeared to allow me to run the command.
When I did this… I received a lot of messages like the following.
Not sure what this means… though
goroutine 111 [select, 2 minutes]:
github.com/openziti/foundation/metrics.(*usageRegistryImpl).run(0xc000e40c60, 0x0, 0x0)
github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:114 +0x132
created by github.com/openziti/foundation/metrics.(*usageRegistryImpl).StartReporting
github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:50 +0xaf
goroutine 112 [select, 2 minutes]:
github.com/openziti/foundation/metrics.(*usageRegistryImpl).sendMsgs(0xc000e40c60, {0x1b465a0, 0x2b8a868}, 0xc001bfa8a0)
github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:134 +0xaa
created by github.com/openziti/foundation/metrics.(*usageRegistryImpl).StartReporting
github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:51 +0x12f
goroutine 78 [IO wait]:
internal/poll.runtime_pollWait(0x7f7b8860ac78, 0x72)
runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc002e36d00, 0xc000074800, 0x0)
internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(…)
internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc002e36d00)
internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc002e36d00)
net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc002e48498)
net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc002e48498)
net/tcpsock.go:262 +0x3d
crypto/tls.(*listener).Accept(0xc002e484b0)
crypto/tls/tls.go:66 +0x2d
net/http.(*Server).Serve(0xc002e288c0, {0x1b53ac0, 0xc002e484b0})
net/http/server.go:3002 +0x394
net/http.(*Server).ServeTLS(0xc002e288c0, {0x1b564f0, 0xc002e48498}, {0x0, 0x87}, {0x0, 0x0})
net/http/server.go:3074 +0x3fe
net/http.(*Server).ListenAndServeTLS(0xc002e288c0, {0x0, 0x0}, {0x0, 0x0})
net/http/server.go:3229 +0x12f
github.com/openziti/fabric/xweb.(*Server).Start(0xc000888ae0)
github.com/openziti/fabric@v0.17.88/xweb/server.go:193 +0x21e
github.com/openziti/fabric/xweb.(*XwebImpl).Run.func1()
github.com/openziti/fabric@v0.17.88/xweb/web.go:123 +0x45
created by github.com/openziti/fabric/xweb.(*XwebImpl).Run
github.com/openziti/fabric@v0.17.88/xweb/web.go:122 +0x57
goroutine 124 [chan receive, 2 minutes]:
github.com/openziti/fabric/controller/xt.(*failureCosts).CreditOverTime.func1()
github.com/openziti/fabric@v0.17.88/controller/xt/failure.go:29 +0x56
created by github.com/openziti/fabric/controller/xt.(*failureCosts).CreditOverTime
github.com/openziti/fabric@v0.17.88/controller/xt/failure.go:28 +0x90
goroutine 130 [select, 2 minutes]:
github.com/openziti/foundation/metrics.(*intervalCounterImpl).run(0xc000152a80)
github.com/openziti/foundation@v0.17.22/metrics/interval_counter.go:125 +0x15f
created by github.com/openziti/foundation/metrics.newIntervalCounter
github.com/openziti/foundation@v0.17.22/metrics/interval_counter.go:70 +0x30
Ahh… when I stopped the controller… they all disappeared… so they are internal go “stuff” with ziti 
at this time, ziti ps stack doesn't seem to support specifying the pid. the problem here is you are running a quickstart install, which has both the ziti router and ziti controller on the same machine. stopping one or the other would let this command work or run the edge router on a different machine.
We probably should support discovering the proper pid to attach to though. I'll open a github issue
@TheLumberjack It’s my understanding this is already supported.
Here is the syntax for the ziti ps stack command:
Usage:
ziti ps stack [<optional-target>] [flags]
Flags:
-h, --help help for stack
--stack-timeout duration Timeout for stack operation (default 5s)
The [<optional-target>] can be a pid or name
Examples in Linux:
sudo ./ziti ps stack $(pidof ziti-router)
sudo ./ziti ps stack ziti-router
Thanks for the followup…