Controller error handshake failed

Support
1

Hi,

I saw the following error message in my Controller logs today.

Here in the example it only affects one system. The error message on the controller, come from several systems, if not all (Because of NAT, I can not find out if it affects all).


Jun 06 12:03:46 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:55692","time":"2025-06-06T12:03:46.000Z"}
Jun 06 12:04:54 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:59550","time":"2025-06-06T12:04:54.187Z"}
Jun 06 12:05:13 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:35478","time":"2025-06-06T12:05:13.354Z"}
Jun 06 12:05:20 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:46696","time":"2025-06-06T12:05:20.806Z"}
Jun 06 12:05:40 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:60574","time":"2025-06-06T12:05:40.873Z"}
Jun 06 12:06:00 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:47594","time":"2025-06-06T12:06:00.950Z"}
Jun 06 12:06:21 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:47488","time":"2025-06-06T12:06:21.019Z"}
Jun 06 12:06:31 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:47684","time":"2025-06-06T12:06:31.050Z"}
Jun 06 12:06:41 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:37418","time":"2025-06-06T12:06:41.077Z"}
Jun 06 12:07:41 ctrl1 ziti[1135]: {"_context":"tls:0.0.0.0:1280","error":"EOF","file":"github/openziti/transport/v2@v2.0.167/tls/listener.go:260","func":"github/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"152.53.X.X:32880","time":"2025-06-06T12:07:41.362Z"}

On the affected server, however, I can only find the following log entry

Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]   ERROR tlsuv:engine.c:907 openssl read: error:0A000126:SSL routines::unexpected eof while reading
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    INFO ziti-sdk:channel.c:924 on_channel_data() ch[0] channel disconnected [-103/software caused connection abort]
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    WARN ziti-sdk:channel.c:849 on_channel_close() ch[0] disconnected from edge router[edge1.ziti.example.com] -103(software caused connection abort)
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1076 on_ziti_event() ztx[mailsrv] router edge1.ziti.example.com disconnected
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    WARN ziti-sdk:bind.c:468 on_message() binding failed: -21/connection to edge router terminated
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    WARN ziti-sdk:bind.c:468 on_message() binding failed: -21/connection to edge router terminated
Jun 06 12:04:53 mail ziti-edge-tunnel[623]: (623)[      498.806]    INFO ziti-sdk:channel.c:834 reconnect_channel() ch[0] reconnecting in 4729ms (attempt = 1)
Jun 06 12:04:58 mail ziti-edge-tunnel[623]: (623)[      503.559]    INFO ziti-sdk:channel.c:734 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.5.4|6739025d0bd6|2025-03-28T13:39:47Z|linux|amd64
Jun 06 12:04:58 mail ziti-edge-tunnel[623]: (623)[      503.559]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1073 on_ziti_event() ztx[mailsrv] router edge1.ziti.example.com connected
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]   ERROR tlsuv:engine.c:907 openssl read: error:0A000126:SSL routines::unexpected eof while reading
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    INFO ziti-sdk:channel.c:924 on_channel_data() ch[1] channel disconnected [-103/software caused connection abort]
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    WARN ziti-sdk:channel.c:849 on_channel_close() ch[1] disconnected from edge router[edge2.ziti.example.com] -103(software caused connection abort)
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1076 on_ziti_event() ztx[mailsrv] router edge2.ziti.example.com disconnected
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    WARN ziti-sdk:bind.c:468 on_message() binding failed: -21/connection to edge router terminated
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    WARN ziti-sdk:bind.c:468 on_message() binding failed: -21/connection to edge router terminated
Jun 06 12:05:20 mail ziti-edge-tunnel[623]: (623)[      526.342]    INFO ziti-sdk:channel.c:834 reconnect_channel() ch[1] reconnecting in 3635ms (attempt = 1)
Jun 06 12:05:24 mail ziti-edge-tunnel[623]: (623)[      530.014]    INFO ziti-sdk:channel.c:734 hello_reply_cb() ch[1] connected. EdgeRouter version: v1.5.4|6739025d0bd6|2025-03-28T13:39:47Z|linux|amd64
Jun 06 12:05:24 mail ziti-edge-tunnel[623]: (623)[      530.014]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1073 on_ziti_event() ztx[mailsrv] router edge2.ziti.example.com connected
2

A [-103/software caused connection abort] message is generally indicative of a certificate verification failure in the TLS negotiation. I would check to make sure the certificates on both sides of the connection, Controller and endpoints, are valid.

You can use openssl to inspect the certificate to check the valid dates, etc.

openssl x509 -noout -text -in 'cerfile.cer';

You may need to specify the format, like --inform pem. The certificate file paths are in the Controller configuration.

3

I forgot the most important part, all clients/servers work perfectly. I only have the errors in the logs.

I have checked the certificates on the controller, they are still valid for a long time.

On one endpoint I could not check the certificates, I only got the error message from openssl:

Unable to load certificate

I checked the certificates within the identities on the endpoint: /opt/openziti/etc/identities
I also saw in the logs that the error has apparently been occurring since day 1, since the controller has been active