From scratch script and HA

Support
1

I used the "from scratch" script
https://raw.githubusercontent.com/dovholuknf/openziti-compose/main/from-scratch.sh
to create a test network.
I removed the "ZITI_EXTERNAL_CA_INTERMEDIATE_NAME" part, using only the "ZITI_ROOT_CA_NAME".
In prevision of an HA setup, I added "--trust-domain "${ZITI_ROOT_OF_TRUST}" in CA creation and I modify server/client certs like this

ziti pki create server \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_CTRL_CA_NAME}" \
  --key-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}" \
  --server-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}-server" \
  --server-name $ZITI_SPIFFE_ID \
  --dns "${ZITI_NETWORK_COMPONENTS_ADDRESSES}" \
  --spiffe-id "controller/${ZITI_SPIFFE_ID}" \
  --ip "${ZITI_NETWORK_COMPONENTS_IPS}"

The controller is working fine in single node mode. I added a couple of routers and I zitified ZAC to test everything.

I tried to install the second node runnning the same script, with appropriate hostname and spidde id, on another host. I copied the "pki/root.ca" folder from the 1st host and I omitted the "ziti pki create ca" command.
The second controlled is able to start, asking for a join command.

Trying to add the 2nd node to the cluster I'm getting on the 2nd controller

ERROR channel/v3.(*UnderlayDispatcher).Run: {error=[unable to validate peer connection, no certs presented matched the CA for this node]} error handling incoming connection, closing connection

And on the 1st controller:

 {"address":"tls:my.second.ziti:8440","error":"error dialing peer tls:my.second.ziti:8440: dial tcp 35.208.18.2:8440: connect: connection refused","file":"github.com/openziti/ziti/controller/raft/mesh/mesh.go:389","func":"github.com/openziti/ziti/controller/raft/mesh.(*impl).Dial","level":"error","msg":"unable to get or connect raft peer channel","time":"2025-02-21T15:33:09.708Z"}

What am I missing?

2

I think it'll be best to use openssl here and try to verify that the controller config has a configured pki that's valid.

My guess is that your controller's cert is not a chain, but is instead a singular leaf.

Starting with openssl, on the second controller, open the config file and look at the identity section at the top and find the 'key', 'cert' and 'ca' fields then issue:

openssl s_client -connect controller.1.here --cert ctrl2.cert --key ctrl2.key --CAfile ctrl2.ca

I expect this fails? After that check the ctrl2.cert and see if it's a chain, not a singular cert.

Hopefully that gives you what you need?

3

pki is still a misterious object to me..
give me an Oracle db and I will be happy :wink:

from ctrl2 to 1

root@ziti-router:/home/ziti/ziti-ha# openssl s_client -connect oci.mydomain:8440 --cert "/home/ziti/ziti-ha/pki/gcp.mydomain-network-components/certs/ziti.network.components-client.cert" --key "/home/ziti/ziti-ha/pki/gcp.mydomain-network-components/keys/ziti.network.components.key" --CAfile "/home/ziti/ziti-ha/pki/gcp.mydomain-network-components/cas.pem"
CONNECTED(00000003)
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci.mydomain-network-components
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci.mydomain-network-components
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 21 12:03:28 2025 GMT; NotAfter: Feb 20 12:04:28 2030 GMT
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci.mydomain-network-components
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 21 12:03:16 2025 GMT; NotAfter: Feb 19 12:04:11 2035 GMT
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 21 12:03:11 2025 GMT; NotAfter: Feb 16 12:04:10 2045 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGDzCCA/egAwIBAgIQQyT8g6YtfQD45nMlLFZSzzANBgkqhkiG9w0BAQsFADBz
MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpOZXRG
b3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMSkwJwYDVQQDEyBvY2kuY2ljdWNpLml0
LW5ldHdvcmstY29tcG9uZW50czAeFw0yNTAyMjExMjAzMjhaFw0zMDAyMjAxMjA0
MjhaMFYxCzAJBgNVBAYTAlVTMRIwEAYDVQQHEwlDaGFybG90dGUxEzARBgNVBAoT
Ck5ldEZvdW5kcnkxEDAOBgNVBAsTB0FEVi1ERVYxDDAKBgNVBAMTA29jaTCCAiIw
DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAO9hhDic1E2SKAMzJQgFfT71rDTG
Tp8OLkbAP0ELLDHyBOSaStXWc9RnyWm2epGqj8xXNO+6wqkcH/zttzvlUJ2wSAHR
oBmc4aKLpBbqkWyybBRJ60vsKM7jI1wBeQnuTNJalLkN5ePNgDaieQxzFGuq6GAW
smdHnIXbg26DYp2QtNOseVQmOBtmrIJr24k/eeZABVzwbxIWpVi64nw55TH/YWMQ
Go8KtSOQSUzfDJTBZbVKKS9AGuQ7mvUKlraK+fFcyLG2P5ijBMpahyDUYd4QMspM
ATsvf05NyOmD9fobTobSRbspd5BLNZhDei9lcA+KW3qa11VfSvqpxLsnP4PLXvnW
rFJlNra9U9QsaSZOwkLkz8N0DdPhode/lr8qFSvqBTdVata+L4//3tjRBN3G8bmS
+s57Q4da0R/jHmhvpBCiFhI6WB9m4jPUFSvZBqJ3Elo9M/sNiNms5Nc3EhYNSmPd
oaTZ7mXMh4uJ7Lm4L6NoyS/FOCqg8jzFif5XSBBHvAfIXJmR7DzvmDJQnY94DKD1
pqQtzUVkFz6wcTiLGBfHuP9KMejfFue/CT4J2l+4dBfYiCveoHpu8/lrt4Lo7Ll1
SF2v3REljslUkYAkPnFvDFREk6mdr5VOY4Hb6N3bARJHf+9g/+Eo1TGRqotLX6Bc
LHslLwPEdzbAbL6XAgMBAAGjgbswgbgwDgYDVR0PAQH/BAQDAgXgMAwGA1UdEwEB
/wQCMAAwHQYDVR0OBBYEFHuxXonDXzwWg27b3YMPV2c5WojVMB8GA1UdIwQYMBaA
FNCCdIeQcVnoFfasY/tZeKJNllHKMFgGA1UdEQRRME+CCWxvY2FsaG9zdIINb2Np
LmNpY3VjaS5pdIcEfwAAAYcEfwAVR4cEWagTm4Yhc3BpZmZlOi8vY2ljdWNpLml0
L2NvbnRyb2xsZXIvb2NpMA0GCSqGSIb3DQEBCwUAA4ICAQADBI6QDTpDwZteh9sk
yLoHtp13N1ECSBLjiVQC0YIY1RGRhQdJCdpovhyIu58vKmE5HFgziAhADG2fKbXQ
NH2ESKUsNiUw2X/NG/JNwq4jBMx6zDGtHtI1sgCnQjCxuhO+cdd9tdrvdkcbRIQE
/4a14dDIVDEKcfjv9z7bE91At2WP5cIZ0KAY/WauZGl6+rCLjefcRP2/TmQuytPz
m0UtQKGCuWX/Jt89xviRyxlbLMFIib1mgV39qzXSmzQoZU2KEz05W1qMWukwv6Vy
9U+IyKz7ifAaLh+WGynjOuybpvfP+VqDB14bzWilSQ7ASu0Og6Yt+TsgKx6p9q1W
McOdCtB2Q7ylNOCIBk3Kuy4J4KvEFv0HW2gydZO56y3nN3YAAWFXBWz+IC5omRv+
OnkWx21ZRCI4QvM3tAXHVKWGS32oWzjs5M0dxsjH92hzo1Z9QeUlnNiwQI5fJXZb
E0N7VvGGs+/KgBQ1nz27vOz+uiRYRbLCKRQuRumK6hu94JlBOWA8Tk7+QlIC1fgm
KjcJ1tUeDrdbehOTZPXnyIKD+Sfra93rJ6dK8Hn51hnat93rcC+0Hu8y+1isa7kD
DZjOYMXzwDQp2RML2/6n9y/eR10moaTrBfKCY2YJcHr1QLKtIJGF+JikYeVYnpTS
0UgnGTy+oQJUvtYIaxHTyRvRJw==
-----END CERTIFICATE-----
subject=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci
issuer=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = oci.mydomain-network-components
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5433 bytes and written 2471 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: DB3B372949E19CC3588C07293CA8CA93113F7E83120E786D49A4E4CA755A34BE
    Session-ID-ctx:
    Resumption PSK: 9FBAC2BA9A61D7C113EA931192F50E84981389042347F521B6BE2364006BF7C7
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 7d 7f a1 f5 b7 b6 0e 96-04 93 49 72 33 a5 e0 98   }.........Ir3...
    0010 - 9b 5a 37 96 93 db 06 bf-9d 95 90 8f 25 97 6d 7a   .Z7.........%.mz
    0020 - b1 7b d4 f3 64 8c ad 55-1a bc 72 3e 54 fd 4d ed   .{..d..U..r>T.M.
    0030 - 1a d4 9f bb 35 a3 ee e9-fa 2e fd c9 45 1b 9e d1   ....5.......E...
    0040 - 55 84 25 bc cd 12 db 17-b1 5e f8 23 1c 63 25 8b   U.%......^.#.c%.
    0050 - 32 ad 7a 99 7f db 8c e1-22 7a 69 e7 19 f6 20 12   2.z....."zi... .
    0060 - 3b 71 4a 7f 30 1a c2 42-ba e2 1a 15 3e 4b e4 c7   ;qJ.0..B....>K..
    0070 - 45 84 63 14 c5 8f fd c4-1f 10 50 c5 16 d1 b0 e3   E.c.......P.....
    0080 - dd 4c 81 8e 0f a0 a6 5a-b2 5a 1c 26 e6 f8 b6 e7   .L.....Z.Z.&....
    0090 - 04 30 ae c2 4d 84 65 c9-a7 6b 46 ea 52 61 cb 00   .0..M.e..kF.Ra..
    00a0 - af fe d5 a5 00 32 e0 3d-2b 36 ef cd 00 2b 0e 18   .....2.=+6...+..
    00b0 - c0 6c 58 bb 98 9f 5a e3-cd d7 2f e9 06 0b bb 54   .lX...Z.../....T
    00c0 - 7f 63 3f 54 a2 dd a5 c8-62 ea 84 ea 0b 0a 7c 0f   .c?T....b.....|.
    00d0 - 6f 6f 08 0c 20 cf 0e 3c-0f 31 f7 9f e7 a6 2e f8   oo.. ..<.1......
    00e0 - 36 8b 19 b4 dc 5b 9c cd-1a 30 8b df 59 68 1e e7   6....[...0..Yh..
    00f0 - 5d a8 5d a5 6c 3f 8d 23-8f a9 c0 7b 6a 7c 7a 67   ].].l?.#...{j|zg
    0100 - 5e f0 09 82 c3 54 b8 b2-44 0f ab 0a 31 ff c8 3b   ^....T..D...1..;
    0110 - 74 99 6a bb 47 b4 e7 ca-61 e8 1c dc 5e 39 f3 2c   t.j.G...a...^9.,
    0120 - eb af e5 cb be d5 3e 45-b3 54 59 16 2e b2 00 7a   ......>E.TY....z
    0130 - ae 1f e5 9f 50 27 12 2c-cd a9 e8 07 a4 7e 78 ca   ....P'.,.....~x.
    0140 - 14 c2 59 f6 7f da 68 7b-45 0c 3d ec f9 57 e7 ed   ..Y...h{E.=..W..
    0150 - 0a 51 3d 3a 25 a2 5b eb-76 a5 54 c3 fc 48 d8 96   .Q=:%.[.v.T..H..
    0160 - ae 8d 1b 50 6e 6c 02 c3-aa ee f7 4b d9 2e 3c ac   ...Pnl.....K..<.
    0170 - 1e c8 8d 97 e7 da 29 4f-ad c2 68 7e 04 dd cd 4f   ......)O..h~...O
    0180 - 6f ad 5a 66 fe d3 8f b9-ea b9 ac fc d3 73 5f 5f   o.Zf.........s__
    0190 - c4 f4 29 11 7b 6d 64 24-3b 32 5d b9 59 b0 e7 c7   ..).{md$;2].Y...
    01a0 - 27 79 5a 0a 4a 2c dc d6-c2 2e 82 ab ef dd 93 0e   'yZ.J,..........
    01b0 - 49 c9 30 ec 5f 23 d6 05-ce f1 85 16 89 b2 8e d2   I.0._#..........
    01c0 - d8 08 bf 4d c8 60 78 22-3e b6 db 03 42 a0 f5 c4   ...M.`x">...B...
    01d0 - e2 49 cd 29 60 60 95 ca-49 16 59 e2 b8 eb 9a 0a   .I.)``..I.Y.....
    01e0 - 27 27 88 b2 2a 94 7a b5-8b 71 0d 78 95 d1 3f 9a   ''..*.z..q.x..?.
    01f0 - e9 23 f9 37 2d ea de 03-67 41 99 c5 18 d8 47 7b   .#.7-...gA....G{
    0200 - 83 1f 6f 54 29 4c 84 9c-95 55 1b a7 a0 61 05 04   ..oT)L...U...a..
    0210 - 38 cc 71 37 f0 d4 03 8d-f7 2b 35 62 41 46 68 bc   8.q7.....+5bAFh.
    0220 - 77 1e 82 4a 82 b7 48 37-5b ce a5 f8 d9 44 a7 cf   w..J..H7[....D..
    0230 - 83 4b 6c 4f 40 57 a0 8c-5e 41 7c d3 b9 c6 b4 43   .KlO@W..^A|....C
    0240 - 85 a0 2e 0c b0 0a f4 6a-44 f5 bb 02 cc c2 94 e1   .......jD.......
    0250 - 67 6c 01 63 19 e4 40 de-15 5c f5 be 85 37 7d d8   gl.c..@..\...7}.
    0260 - 1c 8e 42 65 d9 ae 13 08-f4 87 19 78 cc 88 58 f3   ..Be.......x..X.
    0270 - f8 28 04 c4 99 7c ba 03-6e d2 b2 67 2f f4 f5 b1   .(...|..n..g/...
    0280 - e3 91 9c 5d cd ca f3 95-bf 8c f9 7f 88 6c 19 9e   ...].........l..
    0290 - 9f 1d 05 a4 ff 82 a4 2d-af 4a bd 5b 09 91 94 17   .......-.J.[....
    02a0 - 3f a5 20 5e 37 0b bd c3-9f ef 7c 1f 0d 37 3b e3   ?. ^7.....|..7;.
    02b0 - 99 f5 c4 80 e8 ec 4c 0e-a2 4d 55 88 15 01 68 6d   ......L..MU...hm
    02c0 - 83 ab 89 b9 de e2 0e ee-51 96 e7 8c ad 34 f7 d9   ........Q....4..
    02d0 - d5 fc 34 76 29 86 a2 b2-76 8a ab 92 0c 80 13 81   ..4v)...v.......
    02e0 - 56 7d c0 0b d3 ac 8e ae-54 2d 8b 88 62 00 6c ea   V}......T-..b.l.
    02f0 - 95 c0 4c 5f f5 95 e5 3e-ff a5 31 ee 3b 36 bd c0   ..L_...>..1.;6..
    0300 - 24 76 53 5d f1 ce 66 a1-43 10 1a eb b7 ae b9 55   $vS]..f.C......U
    0310 - 48 6d a0 ac 0f a4 3a 1e-d6 da a7 44 40 17 5d 2a   Hm....:....D@.]*
    0320 - 22 18 27 b4 05 1b 5c 11-0b 0c 33 a4 a4 8d 72 f3   ".'...\...3...r.
    0330 - d8 c5 37 18 1c 25 9e b1-2c a3 44 c9 f3 e7 1d 5f   ..7..%..,.D...._
    0340 - 4c d2 2a 35 c0 32 a3 49-72 dc 5a 61 22 97 a7 9e   L.*5.2.Ir.Za"...
    0350 - 03 49 f9 39 f7 bb 70 b1-16 59 ed fa 8a 1f dc 2e   .I.9..p..Y......
    0360 - f7 f2 e8 ba d5 55 77 47-ec d4 cb ed 58 f2 0f 56   .....UwG....X..V
    0370 - 44 a7 fe e7 52 78 b6 49-24 13 01 8c b3 68 e7 08   D...Rx.I$....h..
    0380 - 5d 00 96 5f 4a ef f4 dd-48 20 40 6e 23 6a 07 9b   ].._J...H @n#j..
    0390 - c8 06 de 2d ea 90 ae 47-08 92 65 e1 45 4d 1a 21   ...-...G..e.EM.!
    03a0 - da db a1 5c 09 b2 da fa-56 76 59 f1 5e c0 3e df   ...\....VvY.^.>.
    03b0 - e9 f4 4e bd 4a 20 8e df-e7 47 6c 35 e0 e4 76 b7   ..N.J ...Gl5..v.
    03c0 - 2f 57 1e 61 bf 16 04 c6-c6 3b b1 8c f4 90 1b f6   /W.a.....;......
    03d0 - cc f1 05 a4 c3 42 99 10-4f 78 32 1b ea 88 16 e4   .....B..Ox2.....
    03e0 - 63 4c 1d a5 71 a4 e9 63-3a 3d 82 38 43 70 34 3b   cL..q..c:=.8Cp4;
    03f0 - 5e e5 1b cb 37 07 66 9c-b3 f1 ce 70 b1 52 a5 e4   ^...7.f....p.R..
    0400 - 9c 67 7d 82 a4 47 3d a0-99 88 a5 0e fb f8 ae 0f   .g}..G=.........
    0410 - 17 19 fc 14 3d 37 0e 30-50 8e 1e 77 4e 3b ab 87   ....=7.0P..wN;..
    0420 - 98 ee dd 3e 46 1c 70 4b-26 6a 4e 48 35 66 4e 3d   ...>F.pK&jNH5fN=
    0430 - 88 46 97 c1 89 79 9f 06-fb e0 0d b8 66 7d ec c1   .F...y......f}..
    0440 - 53 7e e2 4b 29 74 fe 58-8d c1 9a 37 83 33 bc e3   S~.K)t.X...7.3..
    0450 - 99 9a 09 05 a7 5e ba 93-ac a2 52 fd e9 80 c5 14   .....^....R.....
    0460 - c3 4b 4d a0 f0 ab 9f aa-33 f0 07 e4 60 d8 92 df   .KM.....3...`...
    0470 - bb 30 71 0c ce 19 98 0a-e8 77 d5 fe f2 dc a8 29   .0q......w.....)
    0480 - bc f1 31 28 cf 34 a5 9b-27 7c bc 25 65 0e 27 4c   ..1(.4..'|.%e.'L
    0490 - 19 ff f3 88 97 ba 46 95-fd 5f 92 7d 15 92 cb 07   ......F.._.}....
    04a0 - 4f 8f 72 95 a0 bf 61 1f-c1 02 41 ff e3 9b 83 09   O.r...a...A.....
    04b0 - 79 d3 66 d1 c9 fe 83 00-dc b6 cb a4 6e 08 e0 a7   y.f.........n...
    04c0 - 72 3d 88 2c cb 84 44 b0-5b d7 a9 af 58 95 0f 9f   r=.,..D.[...X...
    04d0 - 76 f0 23 8a 64 30 b3 b6-a9 e3 73 fe a5 2d 21 25   v.#.d0....s..-!%
    04e0 - 41 17 c8 29 8d 5b 8b f4-99 7a aa 06 ae d6 97 8a   A..).[...z......
    04f0 - 53 fd 9c d7 db c0 8a 8a-c3 aa ad a2 ed 2b 73 e3   S............+s.
    0500 - d5 6a 3e 94 20 1b b1 6e-e4 f6 4d 83 02 c2 87 af   .j>. ..n..M.....
    0510 - 8f 08 3d d9 5f a2 6a 6d-f8 fa 1d 1c c0 ac 1b 94   ..=._.jm........
    0520 - 26 b1 8b 42 83 bd bc 56-c2 46 a8 7b 7c 3c e0 4d   &..B...V.F.{|<.M
    0530 - 47 8d e1 90 7b 30 54 16-1f 1c aa 9e 24 34 76 a8   G...{0T.....$4v.
    0540 - 4d 42 d2 f0 a6 2e cd c7-68 a4 a4 36 64 92 ed bc   MB......h..6d...
    0550 - 19 4c 82 f6 69 41 bb 61-a4 f7 e3 d9 e7 25 62 86   .L..iA.a.....%b.
    0560 - d2 68 58 de 4c c0 c9 79-df 0b ac 55 5f 8e 3c 59   .hX.L..y...U_.<Y
    0570 - 7a 82 04 06 63 72 15 02-67 fc 72 e1 fb f1 21 fc   z...cr..g.r...!.
    0580 - ba ee 60 67 85 dd 95 25-76 a4 79 47 e8 67 16 71   ..`g...%v.yG.g.q
    0590 - c6 76 e4 68 ef 8e e0 30-19 12 c9 43 eb 48 77 4c   .v.h...0...C.HwL
    05a0 - e5 5c dd 2c 04 21 44 1f-2e 10 a7 ab 17 a3 71 e5   .\.,.!D.......q.
    05b0 - ae c1 1d 41 ec 8a 30 d5-cd 60 29 53 79 36 b9 74   ...A..0..`)Sy6.t
    05c0 - 96 95 9a d5 c7 6e a2 8f-0f 50 27 f4 24 85 b2 0e   .....n...P'.$...
    05d0 - 5b 47 c6 af ae 8b 95 f1-d0 cc e1 7d 7c 83 24 50   [G.........}|.$P
    05e0 - 8d 6f 49 c4 21 25 d8 a9-a7 f6 cb 65 83 da a3 49   .oI.!%.....e...I
    05f0 - c1 04 53 04 54 cd 30 3f-8b a5 ce 72 86 cf 91 44   ..S.T.0?...r...D
    0600 - 72 0c 0b 29 4e 0c 30 38-2b a8 2e 69 50 b6 00 ce   r..)N.08+..iP...
    0610 - 56 fb 3a e5 00 1c 8b 5f-1b 5f 09 87 54 62 c9 ac   V.:...._._..Tb..
    0620 - 04 df 4d 03 80 5b d8 f6-9d 2f 3d df 1a 7f 94 99   ..M..[.../=.....
    0630 - 6d 5b 1e f9 91 00 e5 a7-40 11 c3 52 e6 89 25 29   m[......@..R..%)
    0640 - 80 69 63 ed 33 af 78 1b-c4 93 8c 53 cb 3d 88 ab   .ic.3.x....S.=..
    0650 - 07 62 5c 24 ba                                    .b\$.

    Start Time: 1740168451
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

from ctrl1 to ctrl2

root@ziti-ha:/home/ziti/ziti-ha# openssl s_client -connect gcp.mydomain:8440 --cert "/home/ziti/ziti-ha/pki/oci.mydomain-network-components/certs/ziti.network.components-client.cert" --key "/home/ziti/ziti-ha/pki/oci.mydomain-network-components/keys/ziti.network.components.key" --CAfile "/home/ziti/ziti-ha/pki/oci.mydomain-network-components/cas.pem"
CONNECTED(00000003)
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGDzCCA/egAwIBAgIQZV5qh6f0qT3BXNXiUi+ZODANBgkqhkiG9w0BAQsFADBz
MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpOZXRG
b3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMSkwJwYDVQQDEyBnY3AuY2ljdWNpLml0
LW5ldHdvcmstY29tcG9uZW50czAeFw0yNTAyMjExNDI5NTJaFw0zMDAyMjAxNDMw
NTJaMFYxCzAJBgNVBAYTAlVTMRIwEAYDVQQHEwlDaGFybG90dGUxEzARBgNVBAoT
Ck5ldEZvdW5kcnkxEDAOBgNVBAsTB0FEVi1ERVYxDDAKBgNVBAMTA2djcDCCAiIw
DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAPS7xPE317lzCwTBCRMnLqxCSHKJ
cAb5GLJ7GJLZH1nHjidf+eXVQoB5mYQGsPXOGN9VHdxww23N8CS1yQhCzZCtN88D
ookseWfca412zMURVYKWOQ9Hatoar3XK5VBTZluk2ULvDPmznzAnTzN4/ooWeW+K
iYS3oPbnLQLWM33H5EmZEIowS4KWDsnOZjZf6qsJo7z8A1mJZLnje8+AKBk38AH8
WJ98yeixrLQZWm+UC0DHMQ1NjT+sV7jTzCCFeKhf4mKFQ0Vegga/TJGUUPlOqYQA
fom+0C7geMr/g8z6wkVSI0cQnK9qQ0lY7HM4JMGWRLOTaghk0AduUTA/SG5iJ3bl
WaylANqvTX9jmXSMmQe79gB9HqBoMl5w6riL3PwhpDAgR7T0kz+/4AzmFF8hzOAw
DmcS9iC5C64VOBtRjU9zmR6p5R6AHIhvzxh3bkdMYn5bc1QthkI0GBANb1USpvHp
7ff7HRuPiKvv5mnpHsm4Qjey5up4pAgwkZmhDLFyjC+mKJqtvcbdtgMcpoE9eWoY
BmK/dVlgecC4oSrcNgDEp+RjR7r+2hHf21neOCj28CIJXA5Z+qOwT0W+l58R2CPx
AJBf77ZELdW/Jy7ZD0HmCDcLDzAxPF/sjnoxsTY7o5V0XBv5ROCkBnnKvLK0SERN
c9nD1II5vtWYYHTDAgMBAAGjgbswgbgwDgYDVR0PAQH/BAQDAgXgMAwGA1UdEwEB
/wQCMAAwHQYDVR0OBBYEFAD2JY3lXH/rO9JAWktHWQMX9qpyMB8GA1UdIwQYMBaA
FKW6FqK4X/tJND3rW5q49Co//appMFgGA1UdEQRRME+CCWxvY2FsaG9zdIINZ2Nw
LmNpY3VjaS5pdIcEfwAAAYcEfwAVR4cEI9ASAoYhc3BpZmZlOi8vY2ljdWNpLml0
L2NvbnRyb2xsZXIvZ2NwMA0GCSqGSIb3DQEBCwUAA4ICAQBU/8jFgsrR6A/HY9BT
7UjwreAjVo7hWR5nihFOZ6qFuG3CFrtqCriHRh6IDRdJrBwk8bZxdW5WScBOwx49
cSWtIoAEf6TyeqmyvWgqOVjy+c+n7L5cneRhru4wGSkYDkkoC2vFJ5NcN/SPhzmm
GL/p4H54riqc8UyKtOupYTDGHqEU2ReNM3S4ypLFqMj3DqyQshe9mWwp7tbWdakJ
FgW+w83sCRf7o4BWaOXOTkqVojGi18CRKxcKLQHi9IE7rfkOMDruL6lMHeeYCide
FoEoXbrRD2G4z8pWrTSgyysotNYKdS5PWsiUCoUkHAsPRwmO0NL4ZbevwZQ+T8gX
Q5eSaUhij+FPlawOdaHoX4q4z7xy4wgD2PStw75NLCtlIN4gsOsmtmSu+kHgWfEq
4/ckoDagTZCsBPdHT31upjYQ1sNDiGHesD5UJdxOyh0SSjYFpVGS8ynhOGIkNg/i
39MynMcgSpCpWA9pC3aPZOnQKR7O8c7K2qZKzqnz6K9L8RHUAJUc/tSdg4Afbwxj
8McIS9yi67cdelr4NqVgbEZf6UmVJgPJTT/rBvrM3J3vYbpDO7pif0R2gMALUNA0
jFQdGNAi5CGXQPSblkEfguV+1aZSFJPZOo2neW+6Qr6NSVNy06HVEbXxBWQiO56X
27J9O4cocD2gttlYn7tC54gPOg==
-----END CERTIFICATE-----
subject=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp

issuer=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components

---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5433 bytes and written 2457 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 7CBFD8199ED028982A6912F0F9E5BAEED06D7E241C625DBCAAB58E8376A907EE
    Session-ID-ctx:
    Resumption PSK: D8AA11F2B24DA03C9538614D0D7E91DB5DECEB8782DD8DA67C453B0E11812D56
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 42 98 a7 d8 7a ff 4f c7-34 0f 74 84 c4 cc 2c 09   B...z.O.4.t...,.
    0010 - 95 34 86 34 32 45 89 9d-d6 84 77 6e 1b 79 b2 29   .4.42E....wn.y.)
    0020 - a4 87 66 bd 31 0e a7 39-f8 86 0b fc ab d0 db 9a   ..f.1..9........
    0030 - 74 ea 35 f2 1b ca 12 49-b5 d8 fc e5 6e ef d1 3d   t.5....I....n..=
    0040 - 6f 46 b8 60 c4 80 20 b7-41 7c ff a5 00 78 6f 43   oF.`.. .A|...xoC
    0050 - 38 af 7f e4 7e b5 09 65-8e 33 57 34 54 14 d6 39   8...~..e.3W4T..9
    0060 - de e9 22 0b a1 88 cb 41-80 9d 87 ed 5c 69 79 30   .."....A....\iy0
    0070 - 38 42 b9 50 7c 7c 67 ac-de dd 89 c2 b1 dc 62 07   8B.P||g.......b.
    0080 - bf d8 3e ed a5 35 34 60-9b 2e 1c f2 1e 42 48 4e   ..>..54`.....BHN
    0090 - de 0b a3 af 86 3e f2 7b-aa 3d a4 aa d2 1d 7e 72   .....>.{.=....~r
    00a0 - ba d5 f7 ca f4 16 b1 56-83 80 f2 a0 9d 4e 7c fe   .......V.....N|.
    00b0 - 2c 42 62 bd 15 cb 16 63-15 1c 4c 9e 7c 3d 49 3e   ,Bb....c..L.|=I>
    00c0 - 9c ea 35 ef 0a 78 1b a6-93 99 a7 1b 91 22 59 ef   ..5..x......."Y.
    00d0 - 47 81 62 64 b1 64 52 e3-5f 12 cf 78 33 a4 42 3e   G.bd.dR._..x3.B>
    00e0 - cf ae 2d b7 2c 1b 6b 2f-70 8d 6a 0b 96 5b 3e 89   ..-.,.k/p.j..[>.
    00f0 - 65 ae ea 1a 2b 97 b3 8b-4b bb 49 c1 4f 8c 5a dd   e...+...K.I.O.Z.
    0100 - 33 43 91 e2 3a c0 92 f2-ac ae c3 31 1c e9 ce ed   3C..:......1....
    0110 - f9 ef 90 d3 79 b6 22 69-12 df 50 d4 3b ca 41 e5   ....y."i..P.;.A.
    0120 - 0f 84 a6 27 b9 54 9e ba-ae 8b 1d 37 89 a6 a0 95   ...'.T.....7....
    0130 - 31 45 aa cd 9d 40 78 6a-fd 2a 5e e4 96 cb 1f 08   1E...@xj.*^.....
    0140 - 47 1f f3 ea ba b6 15 df-25 11 5a 83 bb e2 93 37   G.......%.Z....7
    0150 - 2f a4 ba f7 7c 52 8e ba-c0 6f 7e 6e 02 6d 2f 12   /...|R...o~n.m/.
    0160 - 97 1b c8 7e a4 2e f4 37-4b 6a 60 31 ab b0 18 00   ...~...7Kj`1....
    0170 - 81 4c 4f a6 c4 9c 13 3e-a6 fc f0 d8 4d b1 7f a0   .LO....>....M...
    0180 - 06 b0 a5 a2 a1 33 f3 97-53 da f5 93 c6 58 47 bd   .....3..S....XG.
    0190 - 9c 0f de 53 b7 8a 45 a7-de 49 d2 a7 68 a5 32 54   ...S..E..I..h.2T
    01a0 - 94 fa 11 d4 b9 e2 9f 15-5a b2 b8 5f 76 2c 71 92   ........Z.._v,q.
    01b0 - 18 0e d8 b9 30 06 ce 24-82 d4 65 ac db 34 3e 21   ....0..$..e..4>!
    01c0 - 67 f3 59 9a 48 a0 d2 04-63 25 c9 43 23 ab b2 0e   g.Y.H...c%.C#...
    01d0 - 84 56 e4 dc 98 85 7d 48-c3 5c 50 e3 17 87 25 17   .V....}H.\P...%.
    01e0 - 74 3e f9 a7 a3 3c 1f 84-39 04 11 4e da 3f 3b b0   t>...<..9..N.?;.
    01f0 - ea be 18 48 b5 65 c2 43-cf 8d bb 82 e2 e6 7a 4c   ...H.e.C......zL
    0200 - b0 17 3b 56 1a 9b 4e 3b-47 3b 76 7f 6c d0 52 c9   ..;V..N;G;v.l.R.
    0210 - c2 8d d3 3c bb b9 b7 37-ef b2 d9 38 ad b0 b3 a2   ...<...7...8....
    0220 - 67 a9 e4 8a 7b 2c 89 2b-34 6e d0 7e 26 41 e3 e3   g...{,.+4n.~&A..
    0230 - 2a 80 95 66 a4 14 8b e3-91 c0 b2 98 d1 b6 46 94   *..f..........F.
    0240 - 4a 4f d3 12 dc 00 35 50-1b 3c 9b 95 f6 35 32 da   JO....5P.<...52.
    0250 - db 00 be 47 17 ca 0f f7-11 66 1d 1f 5d 69 d2 93   ...G.....f..]i..
    0260 - 9b b3 b7 c8 cd eb 01 29-14 ec 08 e8 2c ec 51 69   .......)....,.Qi
    0270 - 75 c4 75 08 83 7f 3e 46-19 e6 29 42 05 22 5d bf   u.u...>F..)B."].
    0280 - 2d 24 ac 0d d5 a2 dc 5b-73 c1 3c 4f b8 0b 89 3e   -$.....[s.<O...>
    0290 - 8a 89 88 62 59 26 3a b4-ae 3c 8e ba dc 19 75 56   ...bY&:..<....uV
    02a0 - 0c ef d3 79 70 64 f9 0c-90 5c 15 aa 82 9b de 04   ...ypd...\......
    02b0 - 1e a2 0d 1b 9f 05 02 4e-78 00 a7 fd 74 ee 88 ed   .......Nx...t...
    02c0 - fb 5e 8d 21 50 5c 0c 5b-2e bb 1c b4 ab 5a 4e ac   .^.!P\.[.....ZN.
    02d0 - af 13 2b 37 c6 85 2f 82-d4 37 ce 62 a7 32 cc 90   ..+7../..7.b.2..
    02e0 - ef b4 e9 19 1a 69 00 6c-5e 39 af fc 97 f0 f2 41   .....i.l^9.....A
    02f0 - 85 99 1b ad e9 5e 03 8e-2d 26 57 26 fa 62 40 a8   .....^..-&W&.b@.
    0300 - 2e 7c d8 a7 0e 96 80 17-d1 29 f1 f5 0d 6b f2 7a   .|.......)...k.z
    0310 - 27 64 88 91 f6 38 a5 98-34 aa 64 9d 7e ba 1a 09   'd...8..4.d.~...
    0320 - a1 05 6f 1c dd 8c 18 cb-62 78 4e 61 1f ac 61 e3   ..o.....bxNa..a.
    0330 - a1 ce 38 57 d5 b4 48 f5-c9 e5 f5 1c ba d9 3e 8a   ..8W..H.......>.
    0340 - ee 6c 0e 7c 3b b6 49 8a-1d 0b f5 67 60 21 25 ed   .l.|;.I....g`!%.
    0350 - e5 31 58 6a 34 5b 50 22-f2 9e 97 80 51 c4 7c 9a   .1Xj4[P"....Q.|.
    0360 - d8 6b 99 8e f8 e0 bd f3-2c 91 30 84 37 3b 41 51   .k......,.0.7;AQ
    0370 - 3e 59 c8 5d be a0 d6 7e-c8 70 7d 23 4d b7 94 8e   >Y.]...~.p}#M...
    0380 - 60 46 60 e3 b6 9e 82 ed-65 76 be 22 be 12 28 1c   `F`.....ev."..(.
    0390 - bb 6a 90 86 80 3d 5a f3-b1 9c ef 47 7e 06 31 7e   .j...=Z....G~.1~
    03a0 - f1 24 a9 11 63 f9 29 43-a3 85 7b 36 d0 ec e5 30   .$..c.)C..{6...0
    03b0 - ee c6 87 65 02 9f 0e ba-c5 20 22 13 2d 8c a0 0c   ...e..... ".-...
    03c0 - 36 9b f9 bd 4f 50 72 f9-da 29 29 d3 52 36 d4 51   6...OPr..)).R6.Q
    03d0 - 6c 95 b4 fb 23 59 17 da-f5 62 e5 85 f9 e9 fb 22   l...#Y...b....."
    03e0 - 53 dd 41 7a 83 ce b3 8e-ad 2a 99 c9 8a 9b 5f f3   S.Az.....*...._.
    03f0 - 7a df 1f cc cd 49 f9 0a-ba 54 fe 5f 3d dd 4c 63   z....I...T._=.Lc
    0400 - 35 09 53 bb cf 20 1b 03-57 c6 88 22 27 83 7e 04   5.S.. ..W.."'.~.
    0410 - c4 01 27 94 8d 4b 4f d8-66 18 fd 1d 10 4f 1f 27   ..'..KO.f....O.'
    0420 - a8 de 94 9d 6d 4e 84 e1-d3 f4 8e 09 4c 1d bf 41   ....mN......L..A
    0430 - 12 b4 90 8f 7a a3 8c ed-af 64 e7 f0 06 f0 4f fd   ....z....d....O.
    0440 - 77 86 d7 b2 1e c4 68 6a-b2 37 59 03 e9 1d fc d9   w.....hj.7Y.....
    0450 - ca 31 ac d5 d7 3a c8 7c-0a 42 c1 a8 db 22 27 c8   .1...:.|.B..."'.
    0460 - f8 0c f2 1f 0e 5e 78 30-dd 47 7a 15 71 24 40 88   .....^x0.Gz.q$@.
    0470 - da 02 0e 2e 7a 69 27 b1-5e 22 ef c6 65 5d 7b 65   ....zi'.^"..e]{e
    0480 - c2 87 74 06 78 4d c9 33-9a 50 9c 89 62 a9 df c8   ..t.xM.3.P..b...
    0490 - 7f 58 db 4f d6 d4 bb 5f-e1 93 e4 eb 6b 2c 24 75   .X.O..._....k,$u
    04a0 - 8f 48 23 d6 5a 08 1c e6-c3 eb c5 0a ce fb c5 f6   .H#.Z...........
    04b0 - bf 39 b7 9d a7 2e ec c6-c2 87 f8 32 11 ac 85 10   .9.........2....
    04c0 - f7 31 d2 46 a4 54 3c 02-d2 ec 49 93 06 34 f4 71   .1.F.T<...I..4.q
    04d0 - 1a 94 c5 83 80 a4 2c 18-2e 62 02 af cc c4 16 0e   ......,..b......
    04e0 - 91 4b c8 39 08 4e 27 af-1f 61 c6 d0 d8 c3 70 ad   .K.9.N'..a....p.
    04f0 - 5f 06 0f d6 35 99 bc e3-28 cf 85 54 ab 1f fd c4   _...5...(..T....
    0500 - af a0 61 88 a4 13 13 13-21 2a 53 06 5b b7 78 30   ..a.....!*S.[.x0
    0510 - 7a ad ab 9a 61 b0 9a 42-45 44 4b 8b 29 a3 0f 12   z...a..BEDK.)...
    0520 - 24 7a d6 2d 89 ac 15 65-7f ed db ae d1 76 75 07   $z.-...e.....vu.
    0530 - 36 9c b4 eb 28 cc e1 ee-f6 d8 6f 1e 57 ba c5 ac   6...(.....o.W...
    0540 - 33 b6 53 34 69 35 41 94-06 5b 7f eb 4e 3d 35 cc   3.S4i5A..[..N=5.
    0550 - 80 89 a5 76 03 c8 fb 53-30 d1 f3 76 4e a2 12 b2   ...v...S0..vN...
    0560 - db a6 41 bd e7 0a bf 41-c7 b3 93 02 62 89 fb 78   ..A....A....b..x
    0570 - 18 35 58 30 25 b0 ec 67-eb 80 0c eb 7c 38 9b 97   .5X0%..g....|8..
    0580 - f5 6f 9f df ab e2 2e a4-6b e5 0b c2 71 a0 9f 13   .o......k...q...
    0590 - be a3 81 e3 b6 5f 29 79-8b bc de 88 71 6c a9 d3   ....._)y....ql..
    05a0 - ce c9 d4 58 67 f1 ae a5-73 89 19 5d 25 e0 ad fc   ...Xg...s..]%...
    05b0 - fd fc 57 ff 14 0a a0 7e-cb 5a 23 46 c1 5a ad 26   ..W....~.Z#F.Z.&
    05c0 - 43 3c c1 56 7a 85 51 24-ee 37 61 0d 87 1a cf 2a   C<.Vz.Q$.7a....*
    05d0 - 6e 74 8b a2 01 09 0d 86-43 18 db 1c f6 2f 54 18   nt......C..../T.
    05e0 - dd 7b 88 59 24 45 03 c0-9c 44 b1 95 c6 e2 82 60   .{.Y$E...D.....`
    05f0 - f9 72 29 47 44 75 f6 4c-1e 0c 4b 2a 44 cd ee 97   .r)GDu.L..K*D...
    0600 - 0d 4d 58 f1 e9 bc ff 6e-6d 6c e3 92 fe ad 6a 5b   .MX....nml....j[
    0610 - 20 7b 2f 38 6e 86 e3 71-3d 3c a1 29 e2 84 68 88    {/8n..q=<.)..h.
    0620 - 6e bd f9 c4 45 85 36 41-07 45 69 c5 9f 83 ba 8e   n...E.6A.Ei.....
    0630 - ec 0e dc 40 ad 8a 41 f2-d2 29 e2 58 11 eb 80 85   ...@..A..).X....
    0640 - b4 90 0b ff b3 21 83 bd-ee 4b f3 30 4d 2d 2a 2a   .....!...K.0M-**
    0650 - bb be 6c 12 0a                                    ..l..

    Start Time: 1740169026
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
4

pki is a fickle and mysterious beast...

Verify return code: 0 (ok)

is what I was looking for. So both sides are able to correctly connect to each other. Well then I might be stumped. I'll have to ask around for other things to search for. I really expected this to have been "the reason" :slight_smile:

5

Could you check the server_cert from both controllers. Specifically, I'm wondering if the file is a chain (more than one cert) or if there's only one cert in the file. Just the count of certs is fine for now. thx

6

server_certs, for both ctrl and edge, contain 3 certificate sections

7

Mmmm. Ok. That might be something. On the second controller can you take the server_cert and break it into three separate files and run this command on the three parts:

openssl x509 -in third.txt -text | grep -E "Issuer|Subject"

For example from my instance:

C:\temp>openssl x509 -in first.txt -text | grep -E "Issuer|Subject"
        Issuer: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231-intermediate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231 server certificate
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:

C:\temp>openssl x509 -in second.txt -text | grep -E "Issuer|Subject"
        Issuer: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231-root-ca Root CA
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231-intermediate
        Subject Public Key Info:
            X509v3 Subject Key Identifier:

C:\temp>openssl x509 -in third.txt -text | grep -E "Issuer|Subject"
        Issuer: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231-root-ca Root CA
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=ip-172-31-11-231-root-ca Root CA
        Subject Public Key Info:
            X509v3 Subject Key Identifier:

I'd wonder what would happen if you ran two nodes on one machine. Are you able to cluster those? That might be a better place/way to start.

8 
root@ziti-router:~# openssl x509 -in cert1.cert -text | grep -E "Issuer|Subject"
        Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components
        Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:
root@ziti-router:~# openssl x509 -in cert2.cert -text | grep -E "Issuer|Subject"
        Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
        Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = gcp.mydomain-network-components
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:
root@ziti-router:~# openssl x509 -in cert3.cert -text | grep -E "Issuer|Subject"
        Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
        Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = myca.root.ca
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:
9

Thanks... This is a head-scratcher for sure... Would you be willing to try it out on the same computer and see if you can cluster two controllers on one machine first? There's something missing, but I'm not seeing what it might be....

10

I will try in the next days. I will use the hosts file to trick the dns resolution.
The pki creation with the script went so smooth that I was sure the node join would have been slick...
Thanks for the help.

11

All on same host. I put the DNS name of second node in /etc/hosts.
I changed paths on yaml config file and listening ports (+1)
I ran all in foreground as root because of <1024 ports
Here is the complete output. Short answer: same error.

Ctrl1 start

...
[   1.378]    INFO ziti/controller/server.NewController: edge controller instance id: cm7gfvb7v0000oxkhnngkl07o
[   1.378]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   1.381]    INFO ziti/controller/sync_strats.(*InstantStrategy).Initialize: {logSize=[10000] listenerBufferSizes=[1000]} initialized controller router data model
[   1.382]    INFO ziti/controller/sync_strats.(*InstantStrategy).BuildAll.func1: {index=[18]} initialized router data model from db
[   1.382]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
[   1.383]    INFO ziti/controller/server.(*Controller).Run: starting edge
[   1.383]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[10s] maxQueueSize=[1] minWorkers=[1] poolType=[pool.listener.ctrl] maxWorkers=[16]} starting goroutine pool
[   1.384]    INFO ziti/controller/server.(*Controller).checkEdgeInitialized: edge initialized
[   1.385]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[   1.681]    INFO ziti/controller/webapis.(*ZitiAdminConsoleFactory).New: initializing ZAC SPA Handler from ./console
[   1.681] WARNING xweb/v2.getDefault: no defualt handlers were found, using the last handler [Binding: zac, Type: *webapis.GenericHttpHandler] as the default
[   1.683]    INFO ziti/controller/network.(*Network).Run: started
[   1.688]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:443 for server client with APIs: [edge-client edge-oidc]
[   1.688]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:18441 for server management with APIs: [edge-management fabric edge-oidc zac]
[   5.123] WARNING github.com/hashicorp/raft.(*Raft).runFollower: {last-leader-addr=[] last-leader-id=[]} heartbeat timeout reached, starting election
[   5.123]    INFO github.com/hashicorp/raft.(*Raft).runCandidate: {node=[Node at tls:oci.mydomain:8440 [Candidate]] term=[5]} entering candidate state
[   5.123]    INFO github.com/hashicorp/raft.(*Raft).runCandidate: {votesNeeded=[1] refused=[0] term=[5] tally=[1]} pre-vote successful, starting election
[   5.128]    INFO github.com/hashicorp/raft.(*Raft).runCandidate: {tally=[1] term=[5]} election won
[   5.128]    INFO github.com/hashicorp/raft.(*Raft).runLeader: {leader=[Node at tls:oci.mydomain:8440 [Leader]]} entering leader state
[   5.128]    INFO ziti/controller/model.(*ControllerManager).PeersConnected: acting as leader, updating controllers with peers, self: 7d3c0c0dc018a886f7be39950c8b351b386ac10a, peer count: 1, peers:

Ctrl2 start

...
[   1.401]    INFO ziti/controller/server.NewController: edge controller instance id: cm7gfwdcc0000pbkh2hpend9v
[   1.402]    INFO ziti/controller/server.(*Controller).Initialize: initializing edge
[   1.402]    INFO ziti/controller/sync_strats.(*InstantStrategy).Initialize: {logSize=[10000] listenerBufferSizes=[1000]} initialized controller router data model
[   1.480]    INFO ziti/controller/sync_strats.(*InstantStrategy).BuildAll.func1: {index=[0]} initialized router data model from db
[   1.480]    INFO ziti/controller/internal/policy.NewSessionEnforcer: {sessionTimeout=[30m0s] frequency=[5s]} session enforcer configured
[   1.481]    INFO ziti/controller/server.(*Controller).Run: starting edge
[   1.481]    INFO ziti/controller.(*Controller).Run.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] maxWorkers=[16] idleTime=[10s] maxQueueSize=[1] poolType=[pool.listener.ctrl]} starting goroutine pool
[   1.482]    INFO channel/v3.(*UnderlayDispatcher).Run: started
[   1.894]    INFO ziti/controller/webapis.(*ZitiAdminConsoleFactory).New: initializing ZAC SPA Handler from ./console
[   1.894] WARNING xweb/v2.getDefault: no defualt handlers were found, using the last handler [Binding: zac, Type: *webapis.GenericHttpHandler] as the default
[   1.894]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:444 for server client with APIs: [edge-client fabric edge-oidc]
[   1.894]    INFO xweb/v2.(*Server).Start: starting ApiConfig to listen and serve tls on 0.0.0.0:18442 for server management with APIs: [edge-management fabric edge-oidc zac]
[   1.896]    INFO ziti/controller/network.(*Network).Run: started
[   4.300] WARNING github.com/hashicorp/raft.(*Raft).runFollower: no known peers, aborting election
[   4.483] WARNING ziti/controller/server.(*Controller).checkEdgeInitialized: the Ziti Edge has not been initialized, no default admin exists. Add this node to a cluster using 'ziti agent cluster add tls:gcp.mydomain:8441' against an existing cluster member, or if this is the bootstrap node, run 'ziti agent controller init' to configure the default admin and bootstrap the cluster

cluster list

╭─────┬────────────────────────┬───────┬────────┬─────────┬───────────╮
│ ID  │ ADDRESS                │ VOTER │ LEADER │ VERSION │ CONNECTED │
├─────┼────────────────────────┼───────┼────────┼─────────┼───────────┤
│ oci │ tls:oci.mydomain:8440 │ true  │ true   │ v1.3.3  │ true      │
╰─────┴────────────────────────┴───────┴────────┴─────────┴───────────╯

Node added: 1st ctrl

[ 346.565]    INFO github.com/hashicorp/raft.(*Raft).appendConfigurationEntry: {servers=[[[{Suffrage:Voter ID:oci Address:tls:oci.mydomain:8440} {Suffrage:Voter ID:gcp Address:tls:gcp.mydomain:8441}]]] command=[AddVoter] server-id=[gcp] server-addr=[tls:gcp.mydomain:8441]} updating configuration
[ 346.571]    INFO github.com/hashicorp/raft.(*Raft).startStopReplication: {peer=[gcp]} added peer, starting replication
[ 346.571]    INFO ziti/controller/raft/mesh.(*impl).Dial: {address=[tls:gcp.mydomain:8441]} dialing raft peer channel
[ 346.571]    INFO ziti/controller/raft/mesh.(*impl).GetOrConnectPeer: {address=[tls:gcp.mydomain:8441]} establishing new raft peer channel
[ 346.572]    INFO ziti/controller.(*Controller).routerDispatchCallback: {addresses=[[tls:oci.mydomain:8440 tls:gcp.mydomain:8441]] index=[21]} syncing updated ctrl addresses to connected routers
[ 346.593]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[1] idleTime=[1s] poolType=[command_handler] maxQueueSize=[250] minWorkers=[0]} starting goroutine pool
[ 346.594]    INFO ziti/controller/raft/mesh.(*impl).PeerConnected: {peerAddr=[tls:gcp.mydomain:8441] peerId=[gcp]} peer connected
[ 346.594]    INFO ziti/controller/raft/mesh.(*impl).GetOrConnectPeer: {address=[tls:gcp.mydomain:8441] peerId=[gcp]} established new raft peer channel
[ 346.594]    INFO ziti/controller/raft/mesh.(*impl).Dial: {address=[tls:gcp.mydomain:8441] peerId=[gcp]} invoking raft connect on established peer channel
[ 346.594]    INFO ziti/controller/raft/mesh.(*Peer).Connect: {peerId=[gcp] address=[tls:gcp.mydomain:8441]} sending connect msg to raft peer
[ 346.594]    INFO ziti/controller/model.(*ControllerManager).PeersConnected: acting as leader, updating controllers with peers, self: 7d3c0c0dc018a886f7be39950c8b351b386ac10a, peer count: 1, peers: 66608dca2c5427d4e8e89bd63bd8dd705d8dd3a2
[ 346.683]    INFO ziti/controller/raft/mesh.(*impl).PeerDisconnected: {peerId=[gcp] peerAddr=[tls:gcp.mydomain:8441]} peer disconnected
[ 346.902]    INFO ziti/controller/raft/mesh.(*impl).Dial: {address=[tls:gcp.mydomain:8441]} dialing raft peer channel
[ 346.902]    INFO ziti/controller/raft/mesh.(*impl).GetOrConnectPeer: {address=[tls:gcp.mydomain:8441]} establishing new raft peer channel
[ 346.921]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[250] maxWorkers=[1] poolType=[command_handler] minWorkers=[0] idleTime=[1s]} starting goroutine pool
[ 346.921]    INFO ziti/controller/raft/mesh.(*impl).PeerConnected: {peerAddr=[tls:gcp.mydomain:8441] peerId=[gcp]} peer connected
[ 346.921]    INFO ziti/controller/raft/mesh.(*impl).GetOrConnectPeer: {address=[tls:gcp.mydomain:8441] peerId=[gcp]} established new raft peer channel
[ 346.921]    INFO ziti/controller/raft/mesh.(*impl).Dial: {peerId=[gcp] address=[tls:gcp.mydomain:8441]} invoking raft connect on established peer channel
[ 346.921]    INFO ziti/controller/raft/mesh.(*Peer).Connect: {peerId=[gcp] address=[tls:gcp.mydomain:8441]} sending connect msg to raft peer
[ 346.922]    INFO ziti/controller/model.(*ControllerManager).PeersConnected: acting as leader, updating controllers with peers, self: 7d3c0c0dc018a886f7be39950c8b351b386ac10a, peer count: 1, peers: 66608dca2c5427d4e8e89bd63bd8dd705d8dd3a2
[ 346.922]    INFO ziti/controller/raft/mesh.(*impl).PeerDisconnected: {peerId=[gcp] peerAddr=[tls:gcp.mydomain:8441]} peer disconnected
[ 349.572] WARNING github.com/hashicorp/raft.(*Raft).checkLeaderLease: {server-id=[gcp] time=[3.000756024s]} failed to contact
[ 349.572] WARNING github.com/hashicorp/raft.(*Raft).checkLeaderLease: failed to contact quorum of nodes, stepping down
[ 349.572]    INFO github.com/hashicorp/raft.(*Raft).runFollower: {leader-id=[] follower=[Node at tls:oci.mydomain:8440 [Follower]] leader-address=[]} entering follower state
[ 349.572]   ERROR ziti/controller/model.(*ControllerManager).PeersConnected: {error=[leadership lost while committing log] ctrlId=[gcp]} could not create controller during peer(s) connection
[ 349.572]   ERROR ziti/controller/model.(*ControllerManager).PeersDisconnected: {error=[CLUSTER_NO_LEADER: Cluster has no leader, unable to make model updates.]} could not update controller during peer(s) disconnection

Node added: 2nd ctrl

[ 297.174]    INFO ziti/controller/raft/mesh.(*impl).AcceptUnderlay: started
[ 297.175]    INFO ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[250] poolType=[command_handler] minWorkers=[0] maxWorkers=[1] idleTime=[1s]} starting goroutine pool
[ 297.292] WARNING ziti/controller/raft/mesh.(*impl).AcceptUnderlay: exited
[ 297.293]   ERROR channel/v3.(*UnderlayDispatcher).Run: {error=[unable to validate peer connection, no certs presented matched the CA for this node]} error handling incoming connection, closing connection
12

Alright. I was gonna give this a try myself. I'll add this to my todo list and see if I can get it working. I'll be in touch

13

This is the exact procedure I used.
I modified slightly the "from scratch" script adding the spiffe id and I semplified the pki with one root and three intermediate only

### 1st ctrl ###
export ZITI_HOME=/home/ziti/ziti-ha
export ZITI_BIN="${ZITI_HOME}/bin"
export ZITI_PWD=admin
export ZITI_ROOT_OF_TRUST="mydomain.my"
export ZITI_SPIFFE_ID=oci
export ZITI_HOSTNAME="${ZITI_SPIFFE_ID}.${ZITI_ROOT_OF_TRUST}"
export ZITI_PKI="${ZITI_HOME}/pki"
export ZITI_CTRL_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_CTRL_EDGE_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_ROUTER_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_ROOT_CA_NAME="myca.root.ca"
export ZITI_CTRL_CA_NAME="${ZITI_HOSTNAME}-network-components"
export ZITI_EDGE_CA_NAME="${ZITI_HOSTNAME}-edge"
export ZITI_SIGN_CA_NAME="${ZITI_HOSTNAME}-identities"
export ZITI_CTRL_ADVERTISED_PORT=8440
export ZITI_CTRL_EDGE_ADVERTISED_PORT=443
export ZITI_ROUTER_PORT=80
export ZITI_EXTERNAL_IP="$(curl -s eth0.me)"
export ZITI_INTERNAL_IP="$(hostname -I)"
 
### 2nd ctrl ###
export ZITI_HOME=/home/ziti/ziti-ha
export ZITI_BIN="${ZITI_HOME}/bin"
export ZITI_PWD=admin
export ZITI_ROOT_OF_TRUST="mydomain.my"
export ZITI_SPIFFE_ID=gcp
export ZITI_HOSTNAME="${ZITI_SPIFFE_ID}.${ZITI_ROOT_OF_TRUST}"
export ZITI_PKI="${ZITI_HOME}/pki"
export ZITI_CTRL_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_CTRL_EDGE_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_ROUTER_ADVERTISED_ADDRESS="${ZITI_HOSTNAME}"
export ZITI_ROOT_CA_NAME="myca.root.ca"
export ZITI_CTRL_CA_NAME="${ZITI_HOSTNAME}-network-components"
export ZITI_EDGE_CA_NAME="${ZITI_HOSTNAME}-edge"
export ZITI_SIGN_CA_NAME="${ZITI_HOSTNAME}-identities"
export ZITI_CTRL_ADVERTISED_PORT=8440
export ZITI_CTRL_EDGE_ADVERTISED_PORT=443
export ZITI_ROUTER_PORT=80
export ZITI_EXTERNAL_IP="$(curl -s eth0.me)"
export ZITI_INTERNAL_IP="$(hostname -I)"

## only run on 1st ctrl
$ZITI_BIN/ziti pki create ca \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_ROOT_CA_NAME}" \
  --ca-file "${ZITI_ROOT_CA_NAME}" \
  --trust-domain "${ZITI_ROOT_OF_TRUST}"
## only run on 1st ctrl

$ZITI_BIN/ziti pki create intermediate \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_ROOT_CA_NAME}" \
  --intermediate-name "${ZITI_CTRL_CA_NAME}" \
  --intermediate-file "${ZITI_CTRL_CA_NAME}" \
  --max-path-len "1"
  
$ZITI_BIN/ziti pki create intermediate \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_ROOT_CA_NAME}" \
  --intermediate-name "${ZITI_EDGE_CA_NAME}" \
  --intermediate-file "${ZITI_EDGE_CA_NAME}" \
  --max-path-len "1"
  
$ZITI_BIN/ziti pki create intermediate \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_ROOT_CA_NAME}" \
  --intermediate-name "${ZITI_SIGN_CA_NAME}" \
  --intermediate-file "${ZITI_SIGN_CA_NAME}" \
  --max-path-len "1"

### Create Server and Client certs for the control plane and the http api
ZITI_NETWORK_COMPONENTS_PKI_NAME="ziti.network.components"
ZITI_NETWORK_COMPONENTS_ADDRESSES="localhost,${ZITI_HOSTNAME}"
ZITI_NETWORK_COMPONENTS_IPS="127.0.0.1,127.0.21.71,${ZITI_INTERNAL_IP},${ZITI_EXTERNAL_IP}"

$ZITI_BIN/ziti pki create key \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_CTRL_CA_NAME}" \
  --key-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}"

$ZITI_BIN/ziti pki create server \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_CTRL_CA_NAME}" \
  --key-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}" \
  --server-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}-server" \
  --server-name $ZITI_SPIFFE_ID \
  --dns "${ZITI_NETWORK_COMPONENTS_ADDRESSES}" \
  --spiffe-id "controller/${ZITI_SPIFFE_ID}" \
  --ip "${ZITI_NETWORK_COMPONENTS_IPS}"
  
$ZITI_BIN/ziti pki create client \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_CTRL_CA_NAME}" \
  --key-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}" \
  --client-file "${ZITI_NETWORK_COMPONENTS_PKI_NAME}-client" \
  --client-name $ZITI_SPIFFE_ID \
  --spiffe-id "controller/${ZITI_SPIFFE_ID}"

ZITI_EDGE_API_PKI_NAME="ziti.edge.controller"
ZITI_EDGE_API_ADDRESSES="${ZITI_NETWORK_COMPONENTS_ADDRESSES}"
ZITI_EDGE_API_IPS="${ZITI_NETWORK_COMPONENTS_IPS}"

$ZITI_BIN/ziti pki create key \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_EDGE_CA_NAME}" \
  --key-file "${ZITI_EDGE_API_PKI_NAME}"
  
$ZITI_BIN/ziti pki create server \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_EDGE_CA_NAME}" \
  --key-file "${ZITI_EDGE_API_PKI_NAME}" \
  --server-file "${ZITI_EDGE_API_PKI_NAME}-server" \
  --server-name $ZITI_SPIFFE_ID \
  --dns "${ZITI_EDGE_API_ADDRESSES}" \
  --spiffe-id "controller/${ZITI_SPIFFE_ID}" \
  --ip "${ZITI_EDGE_API_IPS}"

$ZITI_BIN/ziti pki create client \
  --pki-root="${ZITI_PKI}" \
  --ca-name "${ZITI_EDGE_CA_NAME}" \
  --key-file "${ZITI_EDGE_API_PKI_NAME}" \
  --client-file "${ZITI_EDGE_API_PKI_NAME}-client" \
  --client-name $ZITI_SPIFFE_ID \
  --spiffe-id "controller/${ZITI_SPIFFE_ID}"

### Emit an OpenZiti Controller Config file
#### Set env vars for the create config to function as expected
export ZITI_PKI_CTRL_KEY="${ZITI_PKI}/${ZITI_CTRL_CA_NAME}/keys/${ZITI_NETWORK_COMPONENTS_PKI_NAME}.key"
export ZITI_PKI_CTRL_SERVER_CERT="${ZITI_PKI}/${ZITI_CTRL_CA_NAME}/certs/${ZITI_NETWORK_COMPONENTS_PKI_NAME}-server.chain.pem"
export ZITI_PKI_CTRL_CERT="${ZITI_PKI}/${ZITI_CTRL_CA_NAME}/certs/${ZITI_NETWORK_COMPONENTS_PKI_NAME}-client.cert"
export ZITI_PKI_CTRL_CA="${ZITI_PKI}/${ZITI_CTRL_CA_NAME}/cas.pem"

export ZITI_PKI_EDGE_KEY="${ZITI_PKI}/${ZITI_EDGE_CA_NAME}/keys/${ZITI_EDGE_API_PKI_NAME}.key"
export ZITI_PKI_EDGE_SERVER_CERT="${ZITI_PKI}/${ZITI_EDGE_CA_NAME}/certs/${ZITI_EDGE_API_PKI_NAME}-server.chain.pem"
export ZITI_PKI_EDGE_CERT="${ZITI_PKI}/${ZITI_EDGE_CA_NAME}/certs/${ZITI_EDGE_API_PKI_NAME}-client.cert"
export ZITI_PKI_EDGE_CA="${ZITI_PKI}/${ZITI_EDGE_CA_NAME}/edge.cas.pem"

ZITI_PKI_SIGNER_NAME="-signer"
export ZITI_PKI_SIGNER_KEY="${ZITI_PKI}/${ZITI_SIGN_CA_NAME}/keys/${ZITI_SIGN_CA_NAME}.key"
export ZITI_PKI_SIGNER_CERT="${ZITI_PKI}/${ZITI_SIGN_CA_NAME}/certs/${ZITI_SIGN_CA_NAME}.chain.pem"

$ZITI_BIN/ziti create config controller >${ZITI_HOME}/${ZITI_HOSTNAME}.yaml

cat "${ZITI_PKI}/${ZITI_ROOT_CA_NAME}/certs/${ZITI_ROOT_CA_NAME}.cert" > "${ZITI_PKI}/${ZITI_HOSTNAME}-network-components/cas.pem"
cp "${ZITI_PKI}/${ZITI_HOSTNAME}-network-components/cas.pem" "${ZITI_PKI}/${ZITI_HOSTNAME}-edge/edge.cas.pem"

mkdir ${ZITI_HOME}/db
$ZITI_BIN/ziti controller edge init "${ZITI_HOME}/${ZITI_HOSTNAME}.yaml" -u "admin" -p $ZITI_PWD
14

It sure feels like some kind of bug. I was able to reproduce the issue myself, locally using docker. I haven't been able to really trace it back but I think I have a workaround for you for now if you want to try. Start your 2 or 3 controllers and get them all up and running.

  • get the intermediate CA from cluster member 2 and cluster member 3 and cat these pems onto controller member 1's ca bundle.
  • restart controller member 1
  • issue controller member 2 join command: ziti agent cluster add tls:controller1.docker.ziti:8440
  • issue controller member 3 join command: ziti agent cluster add tls:controller1.docker.ziti:8440

You'd obviously replace the FQDN with yours. After the controllers join the cluster you should be a able to list them

╭─────────────┬──────────────────────────────────┬───────┬────────┬─────────────────┬───────────╮
│ ID          │ ADDRESS                          │ VOTER │ LEADER │ VERSION         │ CONNECTED │
├─────────────┼──────────────────────────────────┼───────┼────────┼─────────────────┼───────────┤
│ controller1 │ tls:controller1.docker.ziti:8440 │ true  │ false  │ <not connected> │ false     │
│ controller2 │ tls:controller2.docker.ziti:8440 │ true  │ false  │ <not connected> │ false     │
│ controller3 │ tls:controller3.docker.ziti:8440 │ true  │ false  │ v1.3.3          │ true      │
╰─────────────┴──────────────────────────────────┴───────┴────────┴─────────────────┴───────────╯

Hopefully that makes enough sense. I can push my whole project out to github if you want to look at the full monty

I pushed a README, compose file and scripts out to discourse-support/discourse_4005 at main · dovholuknf/discourse-support · GitHub if you want to have a look

15

It took a while :sweat_smile:

I recreated the controllers many times and modified the script to be more like the one provided in the example.
I still have distinct ctrl plane and edge intermediates.

I copied the ctrl plane intermediate cert from ctrl2 to cas.pem on ctrl1.
Then I issued the join command from ctrl 2

Now the cluster is up, both nodes connected, and a leader has been voted.

Now, I guess, I should copy ctrl1 intermediate to cas.pem on ctrl2 too.
And what about edge intermediates?

Adding a third node I should do the same, so on every node the cas.pem is a bundle of the common root CA and every other intermediate.

16

Let's just start off with a reminder that this is almost certainly a bug.. :slight_smile: I plan to talk to the team today about it. I think there's a small issue where only a leaf is used from the 'existing member' during the callback to the 'wannabe joiner'. Assuming I'm right, when that bug is fixed, you won't have to deal with this ...

I did it with three nodes on purpose. Basically every new node you want to to join the cluster, you'll have to cat the joiner's intermediate into the ca bundle of the "current member". In my case, I cat'ed "candidate joiner controller 2" and "candidate joiner controller 3" intermediates into the "main controller 1" ca bundle so that when controller 2 or 3 tried to join, controller 1 was able to connect back to those controllers.

So for right now - you need to cat the intermediate from any prospective joiner onto the ca bundle of the controller you want to issue the join command TO... That make sense?

I'm sure we'll be fixing this issue real soon tho (assuming I'm right about what's happening) :slight_smile:

17

I've spent all day trying to understand how it is that openssl will connect correctly, yet OpenZiti won't.

One thing we can check is what your first controller's cert field looks like. Using your own scripts, can you confirm that after all the configuration is done you end up with an identity block at the top of the controller's config file where the cert fields is not a 'chain'?

For example, the server cert will show as a chain, but the client cert on my locally running instances are not a chain.

Correct Identity block:

identity:
  cert:        "/mnt/persistent/pki/controller1.docker.ziti.intermediate/certs/controller1-client.chain.pem"
  server_cert: "/mnt/persistent/pki/controller1.docker.ziti.intermediate/certs/controller1-server.chain.pem"

Incorrect identity block:

identity:
  cert:        "/mnt/persistent/pki/controller2.docker.ziti.intermediate/certs/controller2-client.cert"
  server_cert: "/mnt/persistent/pki/controller2.docker.ziti.intermediate/certs/controller2-server.chain.pem"

I was able to fix this for my demonstration repo by adding a single env var set before creating the config file:

export ZITI_PKI_CTRL_CERT="${ZITI_PKI}/${ZITI_INTERMEDIATE_NAME}/certs/${ZITI_CTRL_NAME}-client.chain.pem"

So you could likely add this one export as well and you'll be able to make your cluster. I'm going to work on updating the ziti cli so that the config creation uses the chain, not the singular cert and I still want to understand why openssl will verify the connection in this situation.

Cheers

1 Like
18

You nailed it.
My configuration is the first one.

19

I was able to create a three node cluster from scratch with the modified script in 15 minutes.
Thank you very much for the support.

1 Like
20

I'll fix the ziti CLI at some point to use the chain by default. I put a change up to the repo, I'll just need to march it through the process. I'll add a reminder to that issue to come back to leave a comment here when i expect it to merge. Hopefully i'll remember to! :slight_smile: