Invalid Enrollment Token Error

Ziti Overlay
1

Hi OpenZiti Team!

I have recently stumbled over an issue with my OpenZiti deployment within K8s. From time to time it happens that an issued OTT (JWT) is not accepted by the Ziti controller, when I renew the enrollment for an existing identity by calling:

POST /enrollments/{enrollment-id}/refresh

Upon enrollment performed by a ziti-edge-tunnel process the following log is written on the controller:

{\"code\":\"INVALID_ENROLLMENT_TOKEN\",\"error\":\"INVALID_ENROLLMENT_TOKEN: The supplied token is not valid\",\"file\":\"github.com/openziti/ziti/controller/response/responder.go:57\",\"func\":\"github.com/openziti/ziti/controller/response.EdgeResponseMapper.toRestModel\",\"level\":\"debug\",\"msg\":\"returning error to REST API\",\"time\":\"2025-06-25T12:51:42.551Z\"}

Also the ziti-edge-tunnel gives me a similar error:

Jun 25 12:51:42 W-9801 systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-06-25T12:51:42.278)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]    INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-06-25T12:51:42.278)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://client-api.network.xxx.xxx:443] controller initialized
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://client-api.network.xxx.xxx:443] controller initialized
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]   ERROR ziti-sdk:ziti_ctrl.c:524 ctrl_body_cb() ctrl[https://client-api.network.xxx.xxx:443] API request[/enroll] failed code[INVALID_ENROLLMENT_TOKEN] message[The supplied token is not valid]
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]   ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://client-api.network.xxx.xxx:443 INVALID_ENROLLMENT_TOKEN[The supplied token is not valid] reason[]
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116060]: (116060)[        0.000]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: JWT not accepted by controller(-3)
Jun 25 12:51:42 W-9801 ziti-edge-tunnel.sh[116056]: ERROR: failed to enroll test.jwt in /opt/openziti/etc/identities
Jun 25 12:51:42 W-9801 systemd[1]: ziti-edge-tunnel.service: Control process exited, code=exited, status=1/FAILURE

This happens although the OTT (JWT) was not yet used and has also not expired.

My JWT looks like:

{
  "alg": "RS256",
  "kid": "xxxxx",
  "typ": "JWT"
}
{
  "iss": "https://client-api.network.xxx.xxx:443",
  "sub": "YwQHOgkAuj",
  "aud": [
    ""
  ],
  "exp": 1750857615, // 2025-06-25T13:05:24+00:00
  "jti": "xxxxxxxxxxx",
  "em": "ott",
  "ctrls": [
    "tls:ctrlplane.network.xxx.xxx:443"
  ]
}

The version I am using are:
Controller: v1.5.4
ziti-edge-tunnel: v1.6.1

The strange thing about this behavior is that this does not happen constantly but instead very sporadically and I have not found yet a way how to reliably reproduce that :confused:

Could you help me further tracking down this issue ?

2

This looks like it might be this issue: Refresh JWTs can desync internal validation token · Issue #3059 · openziti/ziti · GitHub

Which was released in v1.6.2 which is marked pre-release.

To work around this, you can delete the enrollment and recreate it. Or delete the identity and recreate it.

1 Like
3

Hi @andrew.martinez,
thanks for the hint!

I have now been able to successfully reproduce the issue.
But my reproduction steps were not the exact same as stated in your GitHub issue.

Instead of immediately refreshing the JWT after creating an OTT enrollment for the identity I had to wait until the enrollment was expired. Only after refreshing the JWT after the associated enrollment was expired I encountered the described enrollment failure.

In case I was refreshing within the validity period everything was working fine.

Could this be a different issue then ? Or do you think that the implemented fix is also solving that behavior ?

Best Regards
Jan

4

In case you wish to upgrade to an OpenZiti pre-release like 1.6.2 or 1.6.3 to try reproducing the issue, you can override the Helm Chart's default app version by setting input value image.tag=1.6.3.

Be aware that, if the new OpenZiti version entails any database migrations, it may be impossible to roll back without a database snapshot.