Error when trying to create cluster on debian12

Support
1

Hi.
Trying to create ziti cluster hosted on debian12. All ziti-packets are installed in official way using ubuntu jammy repo.
When not in cluster everything works just fine, routers are routing, tunnelers are tunneling and such.
But when i try add another controller to cluster, command (run on new empty controller with certificates generated from root CA of main controller)

ziti agent cluster add tls:other.peer.mydomain:1280

returns error:

Error: cluster add failed: error dialing peer tls:other.peer.mydomain:1280: remote error: tls: error decrypting message

And logs on main controller shows

ZitiController ziti[51723]: [39800.169] DEBUG transport/v2/tls.(*sharedListener).getConfig [tls:0.0.0.0:1280]: {client=[new_node_IP:44006]} client requesting protocols = [ziti-ctrl]
ZitiController ziti[51723]: [39800.169] DEBUG transport/v2/tls.(*sharedListener).getConfig [tls:0.0.0.0:1280]: {client=[new_node_IP:44006]} found handler for proto[ ]
ZitiController ziti[51723]: [39800.194] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[tls: invalid signature by the client certificate: crypto/rsa: verification error] remote=[new_node_IP:44006]} handshake failed

Certificates are created as follows (like in Controller Certificates | OpenZiti manual):

  1. Root CA is created
  2. Intermediate cert is issued by root
  3. Server and client certs are issued by intermediate
    Steps 2 and 3 are repeated for each cluster node.

Would be grateful for pointing out where to look for the cause of the problem.

2

Hi @poppler

Thanks for trying out the clustering mode. Can you post the identity sections from your controller configs, please? As an alternative, you could check and if see how you are referencing the generated certs and keys lines up with the config file here: ziti/doc/ha/ctrl1.yml at main · openziti/ziti · GitHub

Note that the config above only defines cert and key, but not server_cert and server_key. This is an optional setup which uses the server cert for both client and server side. Most people will define both explicitly.

It can be easy to point your config to the wrong thing.

Let me know what you find,
Paul

3

Not OP but running into the same thing…

configs:
ctrl1

identity:
  cert:        "pki/ctrl1/certs/client.chain.pem"
  server_cert: "pki/ctrl1/certs/server.chain.pem"
  key:         "pki/ctrl1/keys/server.key"
  ca:          "pki/ca/certs/ca.cert"

ctrl2

identity:
  cert:        "pki/ctrl2/certs/client.chain.pem"
  server_cert: "pki/ctrl2/certs/server.chain.pem"
  key:         "pki/ctrl2/keys/server.key"
  ca:          "pki/ca/certs/ca.cert"

Per your link I also tried

ca: pki/ctrl<node>/certs/ctrl<node>.chain.pem

on both hosts but get the same error

cluster add failed: unable to dial tls:ctrl2.ziti.domain.com:1280: remote error: tls: error decrypting message

[tls: invalid signature by the client certificate: crypto/rsa: verification error]} handshake failed
for giggles I also tried using ctrl1.chain.pem as ca on both with no change.