I have deployed OpenZiti using this guide: Host OpenZiti Anywhere | OpenZiti
During my testing phase, I deployed and deleted multiple terminators, currently I have 5 active. However, when I go into a service, I can see 10 terminators:
How can I find which terminators are the active ones so I can remove the others?
I hate starting out like this, but I assume you’re creating terminators manually for a good reason? I only ask because in later versions of ziti, I personally have not had to create a terminator manually for a very long time, instead, I use routers/tunnelers and identities and bind the services via policy. Doing that, the system will make terminators for me…
Now I only start with that comment because a.) maybe that’s not obvious? b.) it means you can stop all your identities/routers and discover if there are any terminators that are used by a given identity.
Now that said, I actually don’t know how (or if) you can determine which ones are “active”. @plorenz, I think you’re going to have to comment on this? It’s a part of the fabric I’m just not super-familiar with…
I am actually doing it like this. I created the tunnelers and assigned the "server" tag, then I created the service with these commands:
ziti edge create config ppc.host host.v1 '{"protocol":"tcp", "address":"example.com", "port":'443'}'
ziti edge create config ppc.intercept intercept.v1 '{"protocols":["tcp"],"addresses":["example.com"], "portRanges":[{"low":'443', "high":'443'}, {"low":'80', "high":'80'}]}'
ziti edge create service ppc.service --configs "ppc.host,ppc.intercept"
ziti edge create service-policy ppc.bind Bind --service-roles '@ppc.service' --identity-roles '#server'
ziti edge create service-policy ppc.dial Dial --service-roles '@ppc.service' --identity-roles '#client'
I can see from the endpoints that those are from SDK applications. They should clean themselves up automatically, just as they are created automatically. So my guess is that one of two things is happening.
Let me know if either of those is correct. If not, we’ll do some more digging into how your system got into the current state.
Cheers,
Paul
I gather that's cause they are edge? That makes me wonder if the UI should even display edge terminators in that particular screen. Maybe it's confusing?
I think that could be better, for sure. Maybe we want to show non-edge/tunnel terminators and then a different display of edge terminators? Would also be nice to show who created the terminator using the terminator HostId.
I’m sorry, I think I got lost a little bit in the conversation.
I only have 1 router and 1 controller (running on the same VM). Then I create tunnelers and use identities to say if that tunneler will be “hosting” a service or if the tunneler will be connecting to that service.
My question to this is because I am using “random” for the terminator strategy and I want to make sure that I am not sending traffic to inactive tunnelers
The overlay network actually takes care of that for you. You can't send traffic to an inactive terminator. It does make me wonder though, why you are asking. Are you experiencing packet loss or connectivity failures and are suspecting these terminators are causing issues?
I have 1 tunneler that is taking care of all internal applications, all these applications work perfect and very fast.
Then I have 5 tunnelers that shares the load for 150+ users for 4 applications and I notice that the performance is much worse.
CPU and Memory usage is very low on each tunneler and they are all hosted in AWS (t3.medium instance).
Also I only have 1 controller and 1 router, so the internal applications use the same controller and router and they work great.
Doing some troubleshooting I came with those extra terminators that should not be there and I wanted to see if it was related.
While you can't send traffic to an inactive terminator, if something went wrong somewhere and you ended up with an invalid terminator entry on an active router, that terminator could get selected when setting up the circuit.
However, if that happened, the router would report back to the controller that the terminator was invalid, and if it was edge terminator, it would get cleaned up. The circuit setup would then get retried and as long as there weren't more invalid terminators, the circuit setup would succeed. So even if something goes wrong, it usually fixes itself relatively quickly.
I am not sure what that means but I have more services than the ones that appear there. Also, there are duplicated services there that are not duplicated in the console GUI
It’s not a full list, by default it only shows 10 entries. You’ll see services repeated if you’ve got multiple terminators for the service.
Let’s try ziti fabric ls terminators 'service.name = "ppc.service" limit none'
I'm assuming that for the internal applications, the applications, tunneler and router are all close (in network terms). For the other 4 applications, how is the network between the router and tunneler and tunneler and application?
Everything is hosted in the same AWS region. The internal applications just have internal IPs while the Public ones are reached over public IP, but they are all instances hosted in the same AWS region.
For connecting to the public ones the tunnelers go over a NAT gateway to exit using the same public IP, for the internal ones the connection is directly or over a transit gateway
Hmmm, I agree that doesn’t look right. Are you using ziti-edge-tunnel?
If you’re not getting connect timeouts, and the number of terminators hasn’t shrunk then I would guess that the tunneler has created multiple valid terminators to the same router, which it’s supposed to do.
It’s also interested that there are 5 identities and 15 terminators, but the terminators aren’t even spread across the identities.
I’m going tag @ekoby and see if he has thoughts on this.
I am catching up on this thread now and I have a few questions:
-j flag for JSON output there will be more information in the outputI just checked the releases – 0.22.0 has a number of fixes wrt terminators, 0.22.1 was just release today. Definitely try the latest release
fyi 0.22.1 is broken if you’re running on ubuntu 20 or later due to missing libssl dependencies. We build the binary on ubuntu 18 which has libssl-1.1, which is not available on newer ubuntu releases. I’m working on creating a build of ziti-edge-tunnel that has fresher libssl dependencies.