SOLVED - Router Enrollment JWT Signature Issue (PKI + ALB Learnings)
TL;DR: We had TWO major issues:
- Outdated Ziti version (v1.1.3) with improper PKI structure
- AWS ALB terminating TLS instead of TCP passthrough
Both needed to be fixed. Here's everything we learned.
Part 1: PKI and Version Issues
The Old Version Problem
We initially downloaded Ziti from https://get.openziti.io/latest/arm64 which gave us v1.1.3 (May 2024) - over a year old! The latest is v1.6.9.
Lesson: Always verify the version you're installing. The "latest" URL may not be current.
The PKI Structure We Needed
After studying nenkoru's manual PKI guide and the from-scratch.sh script, we learned that Ziti expects a multi-level PKI with separate intermediate CAs for different purposes:
Root CA (my-root-ca)
└── External ICA (my-external-ica)
├── Network Components ICA (my-network-ica)
│ ├── network-client.cert (identity.cert)
│ └── network-server.chain.pem (identity.server_cert)
│
├── Edge ICA (my-edge-ica)
│ ├── edge-client.cert (web.identity.cert)
│ └── edge-server.chain.pem (web.identity.server_cert)
│
└── Signing ICA (my-sign-ica)
└── my-sign-ica.chain.pem (edge.enrollment.signingCert)
Using ziti pki Commands
We learned to use Ziti's built-in PKI tools instead of raw OpenSSL:
# Create Root CA
ziti pki create ca --pki-root /opt/ziti/pki --ca-name my-root-ca
# Create Intermediate CAs
ziti pki create intermediate --pki-root /opt/ziti/pki \
--ca-name my-root-ca \
--intermediate-name my-external-ica
ziti pki create intermediate --pki-root /opt/ziti/pki \
--ca-name my-external-ica \
--intermediate-name my-network-ica
# Create server/client certs with proper SANs
ziti pki create server --pki-root /opt/ziti/pki \
--ca-name my-network-ica \
--server-name network-server \
--dns "localhost,ziti.example.com" \
--ip "127.0.0.1"
ziti pki create client --pki-root /opt/ziti/pki \
--ca-name my-network-ica \
--client-name network-client
The Controller Config Structure
We learned the controller config has three distinct identity sections:
# 1. Main identity (for fabric/ctrl plane)
identity:
cert: /opt/ziti/pki/my-network-ica/certs/network-client.cert
server_cert: /opt/ziti/pki/my-network-ica/certs/network-server.chain.pem
key: /opt/ziti/pki/my-network-ica/keys/network-components.key
ca: /opt/ziti/pki/my-network-ica/cas-full.pem
# 2. Signing cert (for signing enrolled identity certs via EST)
edge:
enrollment:
signingCert:
cert: /opt/ziti/pki/my-sign-ica/certs/my-sign-ica.chain.pem
key: /opt/ziti/pki/my-sign-ica/keys/my-sign-ica.key
# 3. Web identity (for edge/management API - THIS IS WHAT SIGNS THE JWT!)
web:
- name: all-apis-localhost
identity:
cert: /opt/ziti/pki/my-edge-ica/certs/edge-client.cert
server_cert: /opt/ziti/pki/my-edge-ica/certs/edge-server.chain.pem
key: /opt/ziti/pki/my-edge-ica/keys/edge-components.key
ca: /opt/ziti/pki/my-edge-ica/edge-cas-full.pem
Key PKI Learnings
- Chain files matter - Use
*.chain.pem for server_cert to include the full chain
- Separate ICAs for separate purposes - Don't use the same cert for everything
cas-full.pem for trust anchors - Include the full CA chain up to root- SANs must match - Server certs need DNS/IP SANs that match how clients connect
signingCert ≠ JWT signing - The signingCert signs enrolled identity certs, NOT the enrollment JWT
Part 2: The ALB Problem (The Final Piece)
Even after fixing all the PKI issues, enrollment still failed with "token signature is invalid."
Source Code Analysis
We cloned the OpenZiti repos and read the actual code:
sdk-golang/ziti/enroll/enroll.go:
func ValidateToken(token *jwt.Token) (interface{}, error) {
cert, err := FetchServerCert(claims.Issuer) // Gets TLS cert from connection
claims.SignatureCert = cert
return cert.PublicKey, nil // Verifies JWT with this cert's public key!
}
ziti/controller/env/appenv.go:
func (ae *AppEnv) GetEnrollmentJwtSigner() (jwtsigner.Signer, error) {
enrollmentCert, err := ae.getEnrollmentTlsCert() // Gets web identity TLS cert
kid := fmt.Sprintf("%x", sha1.Sum(enrollmentCert.Certificate[0]))
return jwtsigner.New(signMethod, enrollmentCert.PrivateKey, kid), nil
}
The Discovery
The enrollment JWT is signed using the web identity's TLS certificate private key. The router verifies it by:
- Connecting to the controller URL
- Getting the TLS certificate from the connection
- Using that cert's public key to verify the JWT
Our AWS ALB was terminating TLS with an ACM certificate, so the router got Amazon's cert instead of our controller's cert!
The Fix
Changed from ALB (Application Load Balancer with HTTPS termination) to NLB (Network Load Balancer with TCP passthrough):
resource "aws_lb" "ziti" {
load_balancer_type = "network" # NOT "application"
}
resource "aws_lb_listener" "tcp" {
protocol = "TCP" # NOT "HTTPS"
# No certificate_arn - passthrough!
}
Router enrollment succeeded immediately after this change.
Summary of All Learnings
| Issue |
Learning |
| Old version |
Always verify Ziti version - don't trust "latest" URLs |
| PKI structure |
Use multi-level ICAs: Network, Edge, Signing |
| Cert generation |
Use ziti pki commands, not raw OpenSSL |
| Chain files |
Always use *.chain.pem for server_cert |
signingCert |
Signs enrolled certs, NOT the enrollment JWT |
| JWT signing |
Done by web.identity TLS cert's private key |
| Load balancers |
MUST use TCP passthrough, not TLS termination |
Deployment Requirements
For enrollment to work, the controller's TLS certificate must be presented directly to clients.
Works: NLB (TCP), direct exposure, nginx stream mode, HAProxy TCP mode
Breaks: ALB (HTTPS), any L7 proxy with TLS termination, Cloudflare proxy mode
Thanks again to @TheLumberjack for the guidance. The PKI guide was essential - it led us down the right path and eventually to reading the source code where we found the final piece of the puzzle!
Thanks for summarizing!