Hi OpenZiti Team!
I have recently stumbled over an issue with my OpenZiti deployment within K8s. From time to time it happens that an issued OTT (JWT) is not accepted by the Ziti controller, when I renew the enrollment for an existing identity by calling:
POST /enrollments/{enrollment-id}/refresh
Upon enrollment performed by a ziti-edge-tunnel process the following log is written on the controller:
{\"code\":\"INVALID_ENROLLMENT_TOKEN\",\"error\":\"INVALID_ENROLLMENT_TOKEN: The supplied token is not valid\",\"file\":\"github.com/openziti/ziti/controller/response/responder.go:57\",\"func\":\"github.com/openziti/ziti/controller/response.EdgeResponseMapper.toRestModel\",\"level\":\"debug\",\"msg\":\"returning error to REST API\",\"time\":\"2025-06-25T12:51:42.551Z\"}
Also the ziti-edge-tunnel gives me a similar error:
Jun 25 12:51:42 W-9801 systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-06-25T12:51:42.278)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-06-25T12:51:42.278)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://client-api.network.xxx.xxx:443] controller initialized
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://client-api.network.xxx.xxx:443] controller initialized
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] ERROR ziti-sdk:ziti_ctrl.c:524 ctrl_body_cb() ctrl[https://client-api.network.xxx.xxx:443] API request[/enroll] failed code[INVALID_ENROLLMENT_TOKEN] message[The supplied token is not valid]
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://client-api.network.xxx.xxx:443 INVALID_ENROLLMENT_TOKEN[The supplied token is not valid] reason[]
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[ 0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: JWT not accepted by controller(-3)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116056]: ERROR: failed to enroll test.jwt in /opt/openziti/etc/identities
Jun 25 12:51:42 W-9801 systemd[1]: ziti-edge-tunnel.service: Control process exited, code=exited, status=1/FAILURE
This happens although the OTT (JWT) was not yet used and has also not expired.
My JWT looks like:
{
"alg": "RS256",
"kid": "xxxxx",
"typ": "JWT"
}
{
"iss": "https://client-api.network.xxx.xxx:443",
"sub": "YwQHOgkAuj",
"aud": [
""
],
"exp": 1750857615, // 2025-06-25T13:05:24+00:00
"jti": "xxxxxxxxxxx",
"em": "ott",
"ctrls": [
"tls:ctrlplane.network.xxx.xxx:443"
]
}
The version I am using are:
Controller: v1.5.4
ziti-edge-tunnel: v1.6.1
The strange thing about this behavior is that this does not happen constantly but instead very sporadically and I have not found yet a way how to reliably reproduce that
Could you help me further tracking down this issue ?