Ziti controller stopped working - Not sure why

Support General Questions
1

My ziti install “just works”. I rarely update it, and I don’t mess with it. I noticed that I do not have any connection today. I reviews the logs, and I see these:

[  82.896]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{mp2nT2n9Eu}->u{classic}->i{mp2nT2n9Eu/YOa0}]: {routerId=[mp2nT2n9Eu] error=[service 1v5IOmbWEg75kIfGutUJW6 has no online terminators for instanceId ] token=[4b252859-453c-4310-8838-d6d3309436ea] operation=[create.circuit]} responded with error
[  83.142]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{mp2nT2n9Eu}->u{classic}->i{mp2nT2n9Eu/YOa0}]: {error=[service 69fuChkD7T3yRKBWQtIkFW has no online terminators for instanceId ] routerId=[mp2nT2n9Eu] operation=[create.circuit] token=[4efb7d01-9110-483f-9ff1-ea1f0acd23e1]} responded with error
[  83.400]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{mp2nT2n9Eu}->u{classic}->i{mp2nT2n9Eu/YOa0}]: {operation=[create.circuit] token=[4b252859-453c-4310-8838-d6d3309436ea] error=[service 1v5IOmbWEg75kIfGutUJW6 has no online terminators for instanceId ] routerId=[mp2nT2n9Eu]} responded with error
[  83.646]   ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{mp2nT2n9Eu}->u{classic}->i{mp2nT2n9Eu/YOa0}]: {token=[4efb7d01-9110-483f-9ff1-ea1f0acd23e1] error=[service 69fuChkD7T3yRKBWQtIkFW has no online terminators for instanceId ] routerId=[mp2nT2n9Eu] operation=[create.circuit]} responded with error

Full log with errors attached. Did some googling, but there are so many error-like entries I was striking out. Thanks!

ziti.txt (51.3 KB)

2

Hey there :waving_hand:

I assume you’re self-hosting your Ziti Controller. I’ll suggest a troubleshooting strategy for the “no terminators” error.

It helps to know how terminators are created when things are working. At least one Ziti Identity must match a Ziti Service Policy (SP) of type “Bind.” That gives the identity permission to host/provide the service. When it starts hosting the service it creates a service terminator on at least one router, more for redundancy if they’re available.

If no identities match a Bind SP, or the allowed identities can’t host the service for some reason (no routers, can’t reach the target, can’t read their identity file(s), etc.), you’ll get the “no terminators” error.

To troubleshoot, go to wherever your identities are located, the ones that match the Bind SP for your Ziti Service. Ensure they’re running normally, and diagnose why they can’t host the service by looking at their logs.

Good luck!

3

I do self-host. And I don’t really understand why - but it just started working the next day.

I’ll keep these notes in case it happens again!

4

Just a follow-up, I wanted to emphasize the point @qrkourier made above, that the most likely cause would be found in the hosting application. If it happens again, check the hosting application logs and if you see anything suspicious we’d be happy to look them over and see if it points to anything.

Thank you,

Paul