While testing my Ziti HA POC i have come across an issue with ZET enrolment.
I have been simulating high load on a single HA controller v1.5.4 to understand how enrollment failure is handled by the ZET client v1.5.12.
If one of my HA Controllers is under high load it's possible that ZET client enrolment will fail. When this happens i see the following logs on ZET.
root@lifeboat-devbox-1:~# ziti-edge-tunnel enroll --jwt ./user3.lifeboat.user.jwt --identity user3.lifeboat.user.json
(28290)[ 0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(28290)[ 0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-05-07T14:22:00.818)
(28290)[ 0.000] INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-05-07T14:22:00.818)
(28290)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] controller initialized
(28290)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] controller initialized
(28290)[ 0.000] WARN ziti-sdk:ziti_ctrl.c:82 code_to_error() unmapped error code: UNHANDLED
(28290)[ 0.000] ERROR ziti-sdk:ziti_ctrl.c:524 ctrl_body_cb() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] API request[/enroll] failed code[UNHANDLED] message[An unhandled error occurred]
(28290)[ 0.000] ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://ziti-controller-1.az.lifeboat.ziti:443 UNHANDLED[An unhandled error occurred] reason[]
(28290)[ 0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: WTF: programming error(-111)
If i wait until the HA controller is no longer under high load then retry enrolment with the same JWT, enrolment fails again with the following different error.
root@lifeboat-devbox-1:~# ziti-edge-tunnel enroll --jwt ./user3.lifeboat.user.jwt --identity user3.lifeboat.user.json
(28393)[ 0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(28393)[ 0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-05-07T14:22:53.876)
(28393)[ 0.000] INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-05-07T14:22:53.876)
(28393)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] controller initialized
(28393)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] controller initialized
(28393)[ 0.000] ERROR ziti-sdk:ziti_ctrl.c:524 ctrl_body_cb() ctrl[https://ziti-controller-1.az.lifeboat.ziti:443] API request[/enroll] failed code[INVALID_ENROLLMENT_TOKEN] message[The supplied token is not valid]
(28393)[ 0.000] ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://ziti-controller-1.az.lifeboat.ziti:443 INVALID_ENROLLMENT_TOKEN[The supplied token is not valid] reason[]
(28393)[ 0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: JWT not accepted by controller(-3)
root@lifeboat-devbox-1:~#
And in the Controller log i see the following.
May 08 07:05:57 ziti-controller-1 ziti[3417]: {"error":"enrollment with id z788dmATpc not found","file":"github.com/openziti/ziti/controller/raft/fsm.go:266","func":"github.com/openziti/ziti/controller/raft.(*BoltDbFsm).Apply","index":13533,"level":"error","msg":"applying log resulted in error","time":"2025-05-08T07:05:57.826Z"}