Edge router starting error

1

Hi I’m getting an error when starting an edge router. The error message does not indicate why it failed:

v0vten@Ursu14-Focal-Mango:~$ ziti-router run ~/.ziti/er2.yaml 
[   0.169]    INFO ziti/ziti/router.run: {routerId=[ptYf1dXbG] revision=[9f08bb3ae2fa] arch=[amd64] build-date=[2023-02-08T02:02:47Z] version=[v0.27.4] configFile=[/home/v0vten/.ziti/er2.yaml] go-version=[go1.19.5] os=[linux]} starting ziti-router
[   0.169]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[0] maxQueueSize=[1000] maxWorkers=[32] idleTime=[30s] poolType=[pool.link.dialer]} starting goroutine pool
[   0.169]    INFO fabric/router/forwarder.(*Faulter).run: started
[   0.169]    INFO fabric/router/forwarder.(*Scanner).run: started
[   0.169]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[0] idleTime=[30s] maxWorkers=[128] poolType=[pool.route.handler] maxQueueSize=[1000]} starting goroutine pool
[   0.169] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[   0.170]    INFO fabric/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":1000000000,"DelayRxStart":false,"WriteTimeout":0}
[   0.170]    INFO fabric/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"MessageQueueSize":10}
[   0.170]    INFO fabric/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[   0.170]    INFO fabric/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[   0.170]    INFO edge/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:3022]} starting channel listener
[   0.170]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.listener.xgress_edge] minWorkers=[1] maxWorkers=[16] idleTime=[10s] maxQueueSize=[1]} starting goroutine pool
[   0.170]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:3022]
[   0.170]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[   0.170]    INFO edge/router/xgress_edge.(*Acceptor).Run: starting
[   5.192]   FATAL ziti/ziti/router.run: {error=[error connecting ctrl (channel synchronization)]} error starting

Below is the yaml file

v: 3

identity:
  cert:                 "/home/v0vten/.ziti/er2.cert"
  server_cert:          "/home/v0vten/.ziti/er2.server.chain.cert"
  key:                  "/home/v0vten/.ziti/er2.key"
  ca:                   "/home/v0vten/.ziti/er2.cas"

ctrl:
  endpoint:             tls:Ursu14-Focal-Kiwi:1280

link:
  dialers:
    - binding: transport
#  listeners:
#    - binding:          transport
#      bind:             tls:0.0.0.0:10080
#      advertise:        tls:Ursu14-Focal-Mango:10080
#      options:
#        outQueueSize:   4

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: Ursu14-Focal-Mango:3022
      connectTimeoutMs: 1000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host #tproxy|host



edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - Ursu14-Focal-Mango
        - localhost
      ip:
        - "127.0.0.1"


#transport:
#  ws:
#    writeTimeout: 10
#    readTimeout: 5
#    idleTimeout: 5
#    pongTimeout: 60
#    pingInterval: 54
#    handshakeTimeout: 10
#    readBufferSize: 4096
#    writeBufferSize: 4096
#    enableCompression: true
#    server_cert: /home/v0vten/.ziti/er2.server.chain.cert
#    key: /home/v0vten/.ziti/er2.key

forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32
2

error connecting ctrl (channel synchronization)

That's one I don't think I've seen before. The first question I would have is: "can the computer running the edge router contact the controller". I would test that using openssl like this looking for any kind of response and not an instant rejection nor a timeout:

openssl s_client -connect Ursu14-Focal-Kiwi:1280

Can the edge router reach the controller?

3

I was able to issue a curl from the router host to get the controller json from the controller host

v0vten@Ursu14-Focal-Mango:~$ curl -sk https://Ursu14-Focal-Kiwi:1280/
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://Ursu14-Focal-Kiwi:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://Ursu14-Focal-Kiwi:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://Ursu14-Focal-Kiwi:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2023-02-08T02:02:47Z","revision":"9f08bb3ae2fa","runtimeVersion":"go1.19.5","version":"v0.27.4"},"meta":{}}
4

And from the router can you use openssl to connec to port 3022 as well?

For example this should succeed (though probably returns a self-signed certificate error, which is expected)

openssl s_client -connect Ursu14-Focal-Kiwi:1280

does this work too?

openssl s_client -connect Ursu14-Focal-Kiwi:3022
5

They both work. I changed 3022 to 6262 though.

v0vten@Ursu14-Focal-Mango:~$ openssl s_client -connect Ursu14-Focal-Kiwi:1280
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGKTCCBBGgAwIBAgIQIFC79nRuUft3RD1t6EpRyzANBgkqhkiG9w0BAQsFADBx
MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpOZXRG
b3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMScwJQYDVQQDEx5VcnN1MTQtRm9jYWwt
S2l3aS1pbnRlcm1lZGlhdGUwHhcNMjMwMjEzMTc1MjM2WhcNMjQwMjEzMTc1MzM1
WjB3MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpO
ZXRGb3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMS0wKwYDVQQDEyRVcnN1MTQtRm9j
YWwtS2l3aSBzZXJ2ZXIgY2VydGlmaWNhdGUwggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQDEp/A0zpwhvPFZpAkkInUXP83I2HMU6BAjzVp1RYkvO4xYaC8w
/WLElwJqfBEdlgD2VRw5g9OjXKC4xx+UhYp6AYy6+2V0MUMFPkNYKdi6IvoY6aKv
Oq7/NkNou/HlMkkj+UCT1eswp832VzJHabm5n1et6Qd8CO6umhyIsmrPgShrGRNU
CZbl0juNYaum8LUjeegDMwGf8c2QQJ6CXe+KMZddIx9QY2L650trltttmqojk8LT
BhexeJg7xQ5vl5b3Ps6Tpa71va2vy+HEmcpnfoqtWT/HuD6pO5LsXjDw3MyXs/kv
iV98wcq85A5BOOZ+cBbRo+tXpDS/6vdfiQ8kijpPmo7qNKae1WcPjsTd9kc5I5cW
aRfPVnGFHO4rPXSmkK3Fa4k3jkV0PMWXWFlEbqQYl0+ISMqNbCgPStv1fGndeBFy
ji7Uc9JdGRwJeKljz/J+ZqkzKZA7ceDzo9WPln+BL28gayD4f+/E2hzBsRtPQzWU
biuOw/VY26lPaX5vLvBm2KRL0JDoJqEwoHvYkD9mhh3ViTwKurQvNdsPv1k4flSj
xTAV6kGqh9zm3LAV2n1+c7mMciADZhC/i4XtgCgbp9DLtQ98KY2anRTEuKHYH56z
hm4aNqzSSe2lXOWcyv7co0TPOlcMro0Lk7Jd7j70sx21cVOFK/l58x1PywIDAQAB
o4G2MIGzMA4GA1UdDwEB/wQEAwIF4DAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTZ
BNgAtR5XK8qB/qWVH6TCE+cKCjAfBgNVHSMEGDAWgBQ3o6smpHRtWEtVc8kgQF8N
3/k9UTBTBgNVHREETDBKghFVcnN1MTQtRm9jYWwtS2l3aYIJbG9jYWxob3N0ghFV
cnN1MTQtRm9jYWwtS2l3aYIRVXJzdTE0LUZvY2FsLUtpd2mHBH8AAAEwDQYJKoZI
hvcNAQELBQADggIBABhoYJtQe716pnGX1J8DiymVcVA98BpJHFuKHZrqU9PPeqTG
PVSi0so0kGc+ZVrfrR+h24EOdTfMkd3g2vh8UBbHcKQjXb4yIszEKtJXiV48/B+N
GPSHmpFpA+tYc1hQr/WIPCEdwTMcnbsOxobsNooHua/dUISG+8E6YJzN7XoiLwOk
ThuIcXxeXIfb27VFuq4mMx8ZXIebjG1/Tfz1jFGcdGNWDllTKSZN8vCIbTrVSus1
S2LSmZFlEzfkorDGxzhmXTlQDeXB3VhPaBt/F7b1t2NTcaGrM55kr5tsF3VYnrzz
eF+rmPDKNRvCsOfehgvtJRxenuXP7dsoktMj8AZBAD8ccjcYFhToocFqHDx7Lpvp
Mmq+lDK1UfwhR+e62/XW507NrGJT+cA+Ir+sMULgkETLardChzpqFs8BsjDvsby/
ovxOPYUgfTYF4q58RR2064+/5deehBxK1/qCxiO/YtL1/0VjdxQN0izf4/vl+q4p
VXwhoSCMzmNaSBmGYoT50m7jCq2JLTucFhIQ/b2jD05QVMj0v5fDOQ0kuDRiCiGI
3YwW320P+ea6/mFgwGnrJFHzrd1ZO6iQnA5wI7eMJzkBjP9KzDiHhyMvT6+1v7a/
nS9qoJxuFjvCcDV9isl8ZlUL91RtlHBqkL446H5+caw1tRvEJfxjm/XHwoWb
-----END CERTIFICATE-----
subject=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate

issuer=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate

---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5449 bytes and written 377 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 77FBF07F0F1A008B47D7857B28FAE9AB2865A70988C5D74189CBD37096CB0386
    Session-ID-ctx: 
    Resumption PSK: F04EBC0E401AB68EE4415366636FB97F772363DCA4FC4F0D52F091BEC5918D85
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - bc d3 8f a9 b2 00 cf e3-d9 77 b2 e1 d2 6e 0e 47   .........w...n.G
    0010 - e5 00 18 c2 b1 c7 c0 31-52 0a ce c8 3e ff 9a 99   .......1R...>...
    0020 - 5a 6d 59 62 63 37 2b c8-a8 25 72 af 51 be 82 85   ZmYbc7+..%r.Q...
    0030 - a3 5c 94 fe c4 19 8f 32-37 b7 ef 30 89 f3 2b a1   .\.....27..0..+.
    0040 - 23 cf 39 1b 46 b7 af 2f-0a 04 4a f9 18 00 97 ab   #.9.F../..J.....
    0050 - 09 09 15 cd a8 a1 1d 48-02 e6 7f d3 d9 27 5b 0d   .......H.....'[.
    0060 - 0d d9 0d 6a 15 aa d9 0b-f2 a0 b0 63 f3 67 73 9f   ...j.......c.gs.
    0070 - 91                                                .

    Start Time: 1676398806
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

v0vten@Ursu14-Focal-Mango:~$ openssl s_client -connect Ursu14-Focal-Kiwi:6262
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
verify return:1
depth=1 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
verify return:1
depth=0 C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate
verify return:1
---
Certificate chain
 0 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
 1 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
 2 s:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
   i:C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-root-ca Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGKTCCBBGgAwIBAgIQIFC79nRuUft3RD1t6EpRyzANBgkqhkiG9w0BAQsFADBx
MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpOZXRG
b3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMScwJQYDVQQDEx5VcnN1MTQtRm9jYWwt
S2l3aS1pbnRlcm1lZGlhdGUwHhcNMjMwMjEzMTc1MjM2WhcNMjQwMjEzMTc1MzM1
WjB3MQswCQYDVQQGEwJVUzESMBAGA1UEBxMJQ2hhcmxvdHRlMRMwEQYDVQQKEwpO
ZXRGb3VuZHJ5MRAwDgYDVQQLEwdBRFYtREVWMS0wKwYDVQQDEyRVcnN1MTQtRm9j
YWwtS2l3aSBzZXJ2ZXIgY2VydGlmaWNhdGUwggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQDEp/A0zpwhvPFZpAkkInUXP83I2HMU6BAjzVp1RYkvO4xYaC8w
/WLElwJqfBEdlgD2VRw5g9OjXKC4xx+UhYp6AYy6+2V0MUMFPkNYKdi6IvoY6aKv
Oq7/NkNou/HlMkkj+UCT1eswp832VzJHabm5n1et6Qd8CO6umhyIsmrPgShrGRNU
CZbl0juNYaum8LUjeegDMwGf8c2QQJ6CXe+KMZddIx9QY2L650trltttmqojk8LT
BhexeJg7xQ5vl5b3Ps6Tpa71va2vy+HEmcpnfoqtWT/HuD6pO5LsXjDw3MyXs/kv
iV98wcq85A5BOOZ+cBbRo+tXpDS/6vdfiQ8kijpPmo7qNKae1WcPjsTd9kc5I5cW
aRfPVnGFHO4rPXSmkK3Fa4k3jkV0PMWXWFlEbqQYl0+ISMqNbCgPStv1fGndeBFy
ji7Uc9JdGRwJeKljz/J+ZqkzKZA7ceDzo9WPln+BL28gayD4f+/E2hzBsRtPQzWU
biuOw/VY26lPaX5vLvBm2KRL0JDoJqEwoHvYkD9mhh3ViTwKurQvNdsPv1k4flSj
xTAV6kGqh9zm3LAV2n1+c7mMciADZhC/i4XtgCgbp9DLtQ98KY2anRTEuKHYH56z
hm4aNqzSSe2lXOWcyv7co0TPOlcMro0Lk7Jd7j70sx21cVOFK/l58x1PywIDAQAB
o4G2MIGzMA4GA1UdDwEB/wQEAwIF4DAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTZ
BNgAtR5XK8qB/qWVH6TCE+cKCjAfBgNVHSMEGDAWgBQ3o6smpHRtWEtVc8kgQF8N
3/k9UTBTBgNVHREETDBKghFVcnN1MTQtRm9jYWwtS2l3aYIJbG9jYWxob3N0ghFV
cnN1MTQtRm9jYWwtS2l3aYIRVXJzdTE0LUZvY2FsLUtpd2mHBH8AAAEwDQYJKoZI
hvcNAQELBQADggIBABhoYJtQe716pnGX1J8DiymVcVA98BpJHFuKHZrqU9PPeqTG
PVSi0so0kGc+ZVrfrR+h24EOdTfMkd3g2vh8UBbHcKQjXb4yIszEKtJXiV48/B+N
GPSHmpFpA+tYc1hQr/WIPCEdwTMcnbsOxobsNooHua/dUISG+8E6YJzN7XoiLwOk
ThuIcXxeXIfb27VFuq4mMx8ZXIebjG1/Tfz1jFGcdGNWDllTKSZN8vCIbTrVSus1
S2LSmZFlEzfkorDGxzhmXTlQDeXB3VhPaBt/F7b1t2NTcaGrM55kr5tsF3VYnrzz
eF+rmPDKNRvCsOfehgvtJRxenuXP7dsoktMj8AZBAD8ccjcYFhToocFqHDx7Lpvp
Mmq+lDK1UfwhR+e62/XW507NrGJT+cA+Ir+sMULgkETLardChzpqFs8BsjDvsby/
ovxOPYUgfTYF4q58RR2064+/5deehBxK1/qCxiO/YtL1/0VjdxQN0izf4/vl+q4p
VXwhoSCMzmNaSBmGYoT50m7jCq2JLTucFhIQ/b2jD05QVMj0v5fDOQ0kuDRiCiGI
3YwW320P+ea6/mFgwGnrJFHzrd1ZO6iQnA5wI7eMJzkBjP9KzDiHhyMvT6+1v7a/
nS9qoJxuFjvCcDV9isl8ZlUL91RtlHBqkL446H5+caw1tRvEJfxjm/XHwoWb
-----END CERTIFICATE-----
subject=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi server certificate

issuer=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = Ursu14-Focal-Kiwi-intermediate

---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5449 bytes and written 377 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
140005049902400:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42
6

6262 is what i meant :slight_smile: thanks for figuring that out… My bad.

Can you run ziti-router run ~/.ziti/er2.yaml -v to run with debug logging and see if any more detail is output?

7

@plorenz was able to spot the problem immediately. In your config it shows:

  endpoint:             tls:Ursu14-Focal-Kiwi:1280

You have pointed the router at the edge/management API (which is REST), you need that to be

  endpoint:             tls:Ursu14-Focal-Kiwi:6262
8

I changed the port to 6262 but it still fails:

v0vten@Ursu14-Focal-Mango:~$ ziti-router run ~/.ziti/er2.yaml -v                                                                

*********************************************************************************

An update with v0.27.5 is available for ziti-router v0.27.4 from 
https://github.com/openziti/ziti/releases/latest/

*********************************************************************************
[   0.560]   DEBUG ziti/ziti/util.LogReleaseVersionCheck: this v0.27.4 build of ziti-router is superseded by v&{v0.27.5 0.27.5 [{https://github.com/openziti/ziti/releases/download/v0.27.5/ziti-darwin-amd64-0.27.5.tar.gz} {https://github.com/openziti/ziti/releases/download/v0.27.5/ziti-linux-amd64-0.27.5.tar.gz} {https://github.com/openziti/ziti/releases/download/v0.27.5/ziti-linux-arm-0.27.5.tar.gz} {https://github.com/openziti/ziti/releases/download/v0.27.5/ziti-linux-arm64-0.27.5.tar.gz} {https://github.com/openziti/ziti/releases/download/v0.27.5/ziti-windows-amd64-0.27.5.zip}]}
[   0.565]    INFO ziti/ziti/router.run: {build-date=[2023-02-08T02:02:47Z] revision=[9f08bb3ae2fa] go-version=[go1.19.5] routerId=[ptYf1dXbG] os=[linux] version=[v0.27.4] configFile=[/home/v0vten/.ziti/er2.yaml] arch=[amd64]} starting ziti-router
[   0.566]    INFO fabric/router/forwarder.(*Faulter).run: started
[   0.566]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[32] poolType=[pool.link.dialer] minWorkers=[0] idleTime=[30s] maxQueueSize=[1000]} starting goroutine pool
[   0.566]    INFO fabric/router/forwarder.(*Scanner).run: started
[   0.566]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxQueueSize=[1000] idleTime=[30s] poolType=[pool.route.handler] minWorkers=[0] maxWorkers=[128]} starting goroutine pool
[   0.566] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[   0.566]   DEBUG edge/router/internal/apiproxy.Start: API Proxy disabled
[   0.567]    INFO fabric/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":1000000000,"DelayRxStart":false,"WriteTimeout":0}
[   0.567]    INFO fabric/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"MessageQueueSize":10}
[   0.567]    INFO fabric/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[   0.567]   DEBUG xweb/v2.RegistryMap.Add: adding xweb factory with binding: health-checks
[   0.567]    INFO fabric/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[   0.567]   DEBUG edge/router/xgress_edge.(*Factory).CreateListener: xgress edge listener options: mtu=65536
randomDrops=false
drop1InN=100
txQueueSize=1
txPortalStartSize=16384
txPortalMaxSize=4194304
txPortalMinSize=16384
txPortalIncreaseThresh=224
txPortalIncreaseScale=1
txPortalRetxThresh=64
txPortalRetxScale=0.75
txPortalDupAckThresh=64
txPortalDupAckScale=0.9
rxBufferSize=4194304
retxStartMs=200
retxScale=1.5
retxAddMs=0
maxCloseWait=30s
getCircuitTimeout=30s
lookupApiSessionTimeout=5s
lookupSessionTimeout=5s
channel.outQueueSize=4
channel.connectTimeout=1s
channel.maxOutstandingConnects=16
channel.maxQueuedConnects=1

[   0.567]    INFO edge/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:3022]} starting channel listener
[   0.567]    INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {poolType=[pool.listener.xgress_edge] minWorkers=[1] maxWorkers=[16] idleTime=[10s] maxQueueSize=[1]} starting goroutine pool
[   0.567]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:3022]
[   0.567]    INFO fabric/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[   0.567]    INFO edge/router/xgress_edge.(*Acceptor).Run: starting
[   0.567]   DEBUG channel/v2.(*reconnectingDialer).Create [tls:Ursu14-Focal-Kiwi:1280]: started
[   0.587]   DEBUG transport/v2/tls.DialWithLocalBinding: server provided [3] certificates
[   0.587]   DEBUG channel/v2.(*reconnectingDialer).sendHello [u{reconnecting}->i{}]: started
[   5.588]   DEBUG channel/v2.ReadV2: received message version bytes: [72 84 84 80]
[   5.588]   DEBUG channel/v2.(*reconnectingDialer).sendHello [u{reconnecting}->i{}]: exited
[   5.588]   DEBUG channel/v2.(*reconnectingDialer).Create [tls:Ursu14-Focal-Kiwi:1280]: exited
[   5.589]   FATAL ziti/ziti/router.run: {error=[error connecting ctrl (channel synchronization)]} error starting
9

Hi Joe,
It’s still connecting to the REST port. This is because of an HA change to handle changing cluster configuration, which unfortunately affected non-HA setups. The bug is fixed in 0.27.5. To fix your setup for now, look for an endpoints file in the same directory as your config file. That will have the incorrect address in it. Please delete the file and restart. Let me know if that fixes the issue.

In 0.27.5+ the endpoints file will only be written if a change to the set of controllers is received from a controller.

Cheers,
Paul

10

Yup, that did it. I used

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/quick/ziti-cli-functions.sh)"; getZiti yes

to get the ziti CLIs. What is the process to update zit? Redo the command?

11

it's not quite so straightforward sadly... Yes you can do the getZiti yes... but the systemd unit files (if you use them) are going to have the path hardcoded into them if you used the quickstart...

Also, just to be sure, i would issue unsetZitiEnv first... so:

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/quick/ziti-cli-functions.sh)"; unsetZitiEnv; getZiti yes