Too many gops-agent process found

Ziti Overlay
1

I am doing some debugging on my gloang reverse proxy.. that could be creating problems with the ziti controller.

when I run the command

ziti ps stack

I receive the following error

error: too many gops-agent process found, including [ziti-controller (pid 3352565), ziti-router (pid 3355101)]

Any tips on what this means or how to fix?

2

I found a tool called gops..

This is what I found when I run it.

860003 1 agent unknown Go version /usr/libexec/oracle-cloud-agent/agent
860056 1 updater unknown Go version /usr/libexec/oracle-cloud-agent/updater/updater
860072 860003 gomon unknown Go version /usr/libexec/oracle-cloud-agent/plugins/gomon/gomon
860177 860164 runcommand unknown Go version /usr/libexec/oracle-cloud-agent/plugins/runcommand/runcommand
3362128 3361365 gops go1.17.2 /root/go/bin/gops
3348505 3348364 rproxy go1.17.2 /home/opc/.ziti/quickstart/instance-20220416-1603/sdk-golang/exercises/http/server/rproxy/rproxy
3352565 1 ziti-controller go1.17.8 /home/opc/.ziti/quickstart/instance-20220416-1603/ziti-bin/ziti-v0.25.4/ziti-controller
3355101 3349256 ziti-router go1.17.8 /home/opc/.ziti/quickstart/instance-20220416-1603/ziti-bin/ziti-v0.25.4/ziti-router

Does this mean that there are too many go apps running on my server?

What I will try is to move the reverse proxy to another server to see if this helps

Let me know if you have any other tips..

3

What I did was to stop the router… this appeared to allow me to run the command.

When I did this… I received a lot of messages like the following.

Not sure what this means… though

goroutine 111 [select, 2 minutes]:

github.com/openziti/foundation/metrics.(*usageRegistryImpl).run(0xc000e40c60, 0x0, 0x0)

github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:114 +0x132

created by github.com/openziti/foundation/metrics.(*usageRegistryImpl).StartReporting

github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:50 +0xaf

goroutine 112 [select, 2 minutes]:

github.com/openziti/foundation/metrics.(*usageRegistryImpl).sendMsgs(0xc000e40c60, {0x1b465a0, 0x2b8a868}, 0xc001bfa8a0)

github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:134 +0xaa

created by github.com/openziti/foundation/metrics.(*usageRegistryImpl).StartReporting

github.com/openziti/foundation@v0.17.22/metrics/usage_registry.go:51 +0x12f

goroutine 78 [IO wait]:

internal/poll.runtime_pollWait(0x7f7b8860ac78, 0x72)

runtime/netpoll.go:234 +0x89

internal/poll.(*pollDesc).wait(0xc002e36d00, 0xc000074800, 0x0)

internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(…)

internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Accept(0xc002e36d00)

internal/poll/fd_unix.go:402 +0x22c

net.(*netFD).accept(0xc002e36d00)

net/fd_unix.go:173 +0x35

net.(*TCPListener).accept(0xc002e48498)

net/tcpsock_posix.go:140 +0x28

net.(*TCPListener).Accept(0xc002e48498)

net/tcpsock.go:262 +0x3d

crypto/tls.(*listener).Accept(0xc002e484b0)

crypto/tls/tls.go:66 +0x2d

net/http.(*Server).Serve(0xc002e288c0, {0x1b53ac0, 0xc002e484b0})

net/http/server.go:3002 +0x394

net/http.(*Server).ServeTLS(0xc002e288c0, {0x1b564f0, 0xc002e48498}, {0x0, 0x87}, {0x0, 0x0})

net/http/server.go:3074 +0x3fe

net/http.(*Server).ListenAndServeTLS(0xc002e288c0, {0x0, 0x0}, {0x0, 0x0})

net/http/server.go:3229 +0x12f

github.com/openziti/fabric/xweb.(*Server).Start(0xc000888ae0)

github.com/openziti/fabric@v0.17.88/xweb/server.go:193 +0x21e

github.com/openziti/fabric/xweb.(*XwebImpl).Run.func1()

github.com/openziti/fabric@v0.17.88/xweb/web.go:123 +0x45

created by github.com/openziti/fabric/xweb.(*XwebImpl).Run

github.com/openziti/fabric@v0.17.88/xweb/web.go:122 +0x57

goroutine 124 [chan receive, 2 minutes]:

github.com/openziti/fabric/controller/xt.(*failureCosts).CreditOverTime.func1()

github.com/openziti/fabric@v0.17.88/controller/xt/failure.go:29 +0x56

created by github.com/openziti/fabric/controller/xt.(*failureCosts).CreditOverTime

github.com/openziti/fabric@v0.17.88/controller/xt/failure.go:28 +0x90

goroutine 130 [select, 2 minutes]:

github.com/openziti/foundation/metrics.(*intervalCounterImpl).run(0xc000152a80)

github.com/openziti/foundation@v0.17.22/metrics/interval_counter.go:125 +0x15f

created by github.com/openziti/foundation/metrics.newIntervalCounter

github.com/openziti/foundation@v0.17.22/metrics/interval_counter.go:70 +0x30

4

Ahh… when I stopped the controller… they all disappeared… so they are internal go “stuff” with ziti :slight_smile:

5

at this time, ziti ps stack doesn't seem to support specifying the pid. the problem here is you are running a quickstart install, which has both the ziti router and ziti controller on the same machine. stopping one or the other would let this command work or run the edge router on a different machine.

We probably should support discovering the proper pid to attach to though. I'll open a github issue

6

filed here: ziti ps stack doesn't allow disambiguation · Issue #732 · openziti/ziti · GitHub

7

@TheLumberjack It’s my understanding this is already supported.

Here is the syntax for the ziti ps stack command:

Usage:
  ziti ps stack [<optional-target>] [flags]

Flags:
  -h, --help                     help for stack
      --stack-timeout duration   Timeout for stack operation (default 5s)

The [<optional-target>] can be a pid or name

Examples in Linux:

sudo ./ziti ps stack $(pidof ziti-router)

sudo ./ziti ps stack ziti-router

1 Like
8

well then - problem solved!!! :slight_smile: thanks @emoscardini

1 Like
9

Thanks for the followup…