When do old terminators get removed?

Support General Questions
1

Hiya, I have a question about Terminators.

I have a HA Ziti system consisting of 3 Controllers and 2 public Edge Routers. I'm running v1.5.0 on the infrastructure and ZET v1.5.4.

In my POC system i am seeing Terminators created but under specific scenarios they are not deleted which seems to result in old/stale terminators remaining on the system seemingly indefinitely.

One way i can reproduce this behaviour is by applying a new piece of configuration on a Controller.

To begin with i have a single ZET client connected with a terminator on one of my 2 Edge Routers.

root@ziti-controller-3:~# ziti edge list terminators 'limit 500'
╭───────────────────────┬──────────────┬───────────────────┬─────────┬───────────────────────┬───────────────────────────┬──────┬────────────┬──────────────╮
│ ID                    │ SERVICE      │ ROUTER            │ BINDING │ ADDRESS               │ IDENTITY                  │ COST │ PRECEDENCE │ DYNAMIC COST │
├───────────────────────┼──────────────┼───────────────────┼─────────┼───────────────────────┼───────────────────────────┼──────┼────────────┼──────────────┤
│ 2Elzi7EeaSVko56JXaKeH │ lifeboat.ssh │ edge-router-detof │ edge    │ 2Elzi7EeaSVko56JXaKeH │ user1.lifeboat.controller │    0 │ default    │            0 │
╰───────────────────────┴──────────────┴───────────────────┴─────────┴───────────────────────┴───────────────────────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1

If i update my host.v1 configuration on a Controller, it triggers an update across the system and new terminators are created.
ziti edge update config ...

Now i have a new terminator for the same identity.

root@ziti-controller-3:~# ziti edge list terminators 'limit 500'
╭───────────────────────┬──────────────┬───────────────────┬─────────┬───────────────────────┬───────────────────────────┬──────┬────────────┬──────────────╮
│ ID                    │ SERVICE      │ ROUTER            │ BINDING │ ADDRESS               │ IDENTITY                  │ COST │ PRECEDENCE │ DYNAMIC COST │
├───────────────────────┼──────────────┼───────────────────┼─────────┼───────────────────────┼───────────────────────────┼──────┼────────────┼──────────────┤
│ 2Elzi7EeaSVko56JXaKeH │ lifeboat.ssh │ edge-router-detof │ edge    │ 2Elzi7EeaSVko56JXaKeH │ user1.lifeboat.controller │    0 │ default    │            0 │
│ Ei67QOcmkbeBzvMbEHYmB │ lifeboat.ssh │ edge-router-detof │ edge    │ Ei67QOcmkbeBzvMbEHYmB │ user1.lifeboat.controller │    0 │ default    │            0 │
╰───────────────────────┴──────────────┴───────────────────┴─────────┴───────────────────────┴───────────────────────────┴──────┴────────────┴──────────────╯
results: 1-2 of 2

The original terminator with ID 2Elzi7EeaSVko56JXaKeH i assume is now stale/old but it never seems to be deleted. Is this a problem ?

2

yes, that wrong.

You stumbled upon a really old bug that was never fixed.

3

Thanks @ekoby.

It seems that these stale/old terminators also occur as a result of a number of other conditions.

For example. If i prevent inbound client connections to my Controllers and Edge routers with a firewall rule for a period of time, then re-open the firewall to allow inbound connections. New terminators are created when the clients resume connection but old terminators remain indefinitely.