K8s router enrollment: failed to parse JWT

Support General Questions
1

Hi,
TL;DR
I am currently trying to roll out the full ziti stack on my OKD cluster in separated namespaces and currently working on the router using this guide.

Error:

+ echo 'INFO: identity secret does not exist, attempting router enrollment'
INFO: identity secret does not exist, attempting router enrollment
+ [[ -n '' ]]
+ ziti router enroll /etc/ziti/config/ziti-router.yaml --jwt /etc/ziti/config/enrollment.jwt --verbose
[   0.000]   DEBUG ziti/ziti/util.LogReleaseVersionCheck: ZITI_CHECK_VERSION is not 'true'. skipping version check
[   0.044]   FATAL ziti/router/enroll.(*RestEnroller).Enroll: {cause=[token signature is invalid: key is of invalid type: RSA verify expects *rsa.PublicKey]} failed to parse JWT

Which relates to this LOC: ziti/router/enroll/enroll.go at baad419e49197c44650826ee42f031f4a58019eb · openziti/ziti · GitHub

Full description:

Currently live (not publicly exposed, only reachable from within the cluster atm):

  • controller
  • console

Missing:

  • router
  • browzer-bootstrapper

My ziti controller is reachable from within the router enrollment job's pod (tested via curl).
I've ran the helm chart from my WSL2 Ubuntu environment using

helm install --debug \
  --namespace demo-ziti-router --generate-name \
  openziti/ziti-router \
    --set-file enrollmentJwt=./router1.jwt \
    --set advertisedHost=router.ziti.demo.local \
    --set ctrl.endpoint=demo-openziti-controller-ctrl.demo-ziti-controller.svc.cluster.local:443 \
    --set persistence.size=1Gi \
    --set edge.advertisedHost=router.ziti.demo.local

The enrollment JWT was generated using the ziti CLI insode of the controller pod: I've (1) copied the JWT string over and (2) copied the jwt file from the pod to my local file system. Both approaches resulted in different file hashes but none of them resulted in the same
{cause=[token signature is invalid: key is of invalid type: RSA verify expects *rsa.PublicKey]} failed to parse JWT error.

CRLF breaks didn't seem to be the issue:

❯ sha256sum router1.jwt && dos2unix router1.jwt && sha256sum router1.jwt
dd3d6854fc2b2e44f173031f0287a8ac27eeaf0aa59268311a8dce2baf81bd15  router1.jwt
dos2unix: converting file router1.jwt to Unix format...
dd3d6854fc2b2e44f173031f0287a8ac27eeaf0aa59268311a8dce2baf81bd15  router1.jwt

Checking the JWT via jwt.io didn't provide any new clues.

JWT (already manually disabled for sharing purposes):

eyJhbGciOiJSUzI1NiIsImtpZCI6ImEzNmU2MjAzYzk0NGJjZmY2NzgwNGI1MjAyZmQzZDljODdlNDFmMzciLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2NvbnRyb2xsZXIueml0aS5qb2luZXkuZGV2OjQ0MyIsInN1YiI6InozcXYxeDF0N0IiLCJhdWQiOlsiIl0sImV4cCI6MTcxMDgxMDIxMCwianRpIjoiZmI4YmIwM2YtYTVmMS00YjY3LWE4YzItNTBhZjU1MmZhZGVjIiwiZW0iOiJlcm90dCIsImN0cmxzIjpudWxsfQ.By7Klk1K6XZo_5VRJMqavl8_b09CiqrtY0JQsP8cvZds9HJFXetpR8PhcIYsF867vV26Hz0pgUNlaZ6EmoLvKs8BvRXAnwXBn0_j4klTGLgHC0Hgc2b6XRx9UACzJ1SmPxiG4p6wB9FvImo4ZWoOUSKxhmeytgO-BCZz3T061TTw_TByF4y8TLk1frgbi6DF8U0Vcg__7pM4BlMhODSSlAWvDmdYNzC_gxcclE7mg3w3WzfrhrLSwkSg8yoot1zXCrPAP9HPCBBGL5URWppP0fnw10au9lQXTs-K1Phz76yrbR3qNresrqFPLEiFQCpxUpcki_pr7TzgXlzCBEO0Bp4aoTU9rooTnEGKOdMGsxQikhWmpyv-mEFjwkxKmMgBnzz_P9VMIOBeVZkn5iSD6pKaTtQqiTI4aO4beip8x6xoJXoteX7ZEr8gby9d7kbEII2Td1pwW4YGpLaScgnc8wcDXAyQsAgNSWduwZ88F45-D9_QL3JvM0uHiGTVdoTqdkwdLXCnveFyt24Ffo2Hp1BT_57xXA8MsU5QhsEuAttWwVW1oLhFHST1Fdv1-eUToNXc1KI4TGnWCRc7-ATOopqfjHtOxJXbONZ7IlRg6W91hMPYqGw-zx7pJ1wvFrOdzbRc_OyLJJYSt0uCGaefXcoduiyHcBPeijbY5LMGGC4

Does any1 have an idea what to check next? I'm asking because I'm kinda frustrated atm.

2 Likes
2

The error indicates that the router's enrollment token (and the TLS server's certificate referenced by the issuer URL) was signed by a key other than an RSA key. I didn't expect that an RSA key would be required. I'll investigate to see if that's a build-time option, a CLI flag, or simply not supported.

Your TLS server cert's pubkey info shows it's an EC key, not RSA, which fits the hypothesis that the enrolling router's ziti version requires RSA.

EDIT: Asked around and confirmed the expectation that EC keys should work. I'm diving deeper to repro and diagnose.

1 Like
3

@elysweyr Contradicting my hypothesis about the router enrollment requiring an RSA signing key, I was able to enroll a router with a token that was signed by the same type of key you're using, EC-P256, assuming the issuer URL in the token is presenting the same certificate publicly as inside the cluster.

I used OpenSSL to inspect the certificate's pubkey info.

openssl s_client -connect 127.0.0.1:1280 <>/dev/null |& openssl x509 -noout -text | grep -A22 'Public Key Info'

The value ctrl.endpoint looks correctly set for the ClusterIP service named demo-openziti-controller-ctrl in the demo-ziti-controller namespace. That's the control plane endpoint provided by the controller and consumed by routers.

The router calls the edge client API advertised inside the JWT to enroll. It fetches that API's server certificate and uses the pubkey to verify the JWT signature.

I assume the router's token was created with ziti edge create edge-router.

I analyzed how we're using the golang-jwt package to parse the token during router enrollment. The signature validation methods are determined by looking at the JWT's signature. This tells me the router's enrollment token must have been signed by an RSA key, which is the default for a Helm-installed Ziti controller.

New hypothesis: the controller is advertising the wrong address.

Diagnosing to confirm or deny this hypothesis: check that the iss property in the router's token is a valid URL for the controller's client API, which is a separate ClusterIP service named like demo-openziti-controller-client.

Fix: Align the controller chart's value named clientApi.advertisedHost and clientApi.advertisedPort with the path the router will use to reach the controller during enrollment. For example, if the router will reach the controller via the ClusterIP service directly, internal to the cluster, then the correct values will resemble these.

clientApi:
    advertisedHost: demo-openziti-controller-ctrl.demo-ziti-controller.svc.cluster.local
    advertisedPort: 443

Upgrading the controller release with these input values will render the controller's YAML config (in a configmap resource). The next router enrollment token will have the new advertised address in the iss claim property.

1 Like
4

Fix: Align the controller chart's value named clientApi.advertisedHost and clientApi.advertisedPort with the path the router will use to reach the controller during enrollment.

Thanks, that did the trick. As I was just testing out the initial setup my client api wasn't exposed yet and the router tried to connect using that external - but not yet available - URL.

1 Like