Certificate renewal showing path //keys/-server.key

kashif · August 20, 2024, 11:42am

Controller: v0.30.0 ZAC: 2.8.7

ziti edge version
RESTY 2024/08/20 11:40:38 ERROR Get "https://ztn:8441/edge/management/v1/version": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-08-20T11:40:38Z is after 2024-08-20T08:26:09Z, Attempt 1

Hi,
I hope you all are doing well.

After executing below 2 commands then I followed the link (Certificate Expired - #5 by TheLumberjack) to renew openziti certificate

export EXTERNAL_DNS="ztn:8443"
source $HOME/.ziti/quickstart/$(hostname -s)/$(hostname -s).env

When I completed and tried to verify values, it showed this
NOTE: IP 192.168.1.32 (LAN/WAN) not showed like it showed in ENTRIES : 127.0.0.1,3.134.108.218 in the link, if not important then please discard this note.

echo "DNS ENTRIES  : $pki_allow_list_dns"
DNS ENTRIES  : ,localhost,ztn,ztn:8443

echo "IP ENTRIES   : $pki_allow_list_ip"
IP ENTRIES   : 127.0.0.1

echo "CTRL KEY FILE: ${ZITI_CTRL_IDENTITY_KEY}"
CTRL KEY FILE: //keys/-server.key

echo "EDGE KEY FILE: ${ZITI_EDGE_CTRL_IDENTITY_KEY}"
EDGE KEY FILE: //keys/-server.key

Which is not showing the full path like

echo ${ZITI_BIN_DIR-}
/root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0

After executing below (not using any ca certificate authority certificate)

root@ztn:~# "${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_CONTROLLER_HOSTNAME} server certificate ${now}"
? Required flag 'pki-root' not specified; Enter PKI Root now: /root/.config/ziti/environments/pki
Using CA name:
error: cannot locate signer: failed fetching bundle  within CA : failed reading CA private key from file /root/.config/ziti/environments/pki/keys/.key: failed reading /root/.config/ziti/environments/pki/keys/.key: open /root/.config/ziti/environments/pki/keys/.key: no such file or directory
root@ztn:~#

Any help please?

Thank you

TheLumberjack · August 20, 2024, 11:47am

Hi @kashif,

My guess is that in the year since that post some of the variables changed. I think it happened around a year ago. I'll get an update command for you and share it here (and there).

TheLumberjack · August 20, 2024, 12:56pm

Hi @kashif, here is an updated set of steps...

Setup some variables for the server

Make sure the end result looks right. You want to make sure you see your control plane and edge api in the DNS ENTRIES list at the end, and you want to make sure the IP ENTRIES is correct as well (if any).

export ZITI_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC:$ZITI_HOME/pki}/${ZITI_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_CTRL_ADVERTISED_ADDRESS}-server.key"
export ZITI_EDGE_CTRL_IDENTITY_KEY=export ZITI_EDGE_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}-server.key"
export ZITI_PKI_OS_SPECIFIC="$HOME/.ziti/quickstart/$(hostname)/pki"
now="$(date '+%Y-%m-%d-%M%S')"

pki_allow_list_dns="localhost"
if [[ "${ZITI_CTRL_ADVERTISED_ADDRESS}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${ZITI_CTRL_ADVERTISED_ADDRESS}"; fi
if [[ "${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}"; fi

if [[ "${EXTERNAL_DNS}" != "" && \
	  ("${EXTERNAL_DNS}" != "${ZITI_CTRL_ADVERTISED_ADDRESS}" && \
	  "${EXTERNAL_DNS}" != "${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}") ]]; then
	  pki_allow_list_dns="${pki_allow_list_dns},${EXTERNAL_DNS}"
fi

pki_allow_list_ip="127.0.0.1"
if [[ "${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}" != "" ]]; then pki_allow_list_ip="${pki_allow_list_ip},${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}"; fi

echo "DNS ENTRIES  : $pki_allow_list_dns"
echo "IP ENTRIES   : $pki_allow_list_ip"
echo "CTRL KEY FILE: ${ZITI_PKI_CTRL_KEY}"
echo "EDGE KEY FILE: ${ZITI_PKI_EDGE_KEY}"

Create a new Server cert for the control plane

export ZITI_CONTROLLER_INTERMEDIATE_NAME="${ZITI_PKI_CTRL_INTERMEDIATE_NAME}"
export ZITI_CONTROLLER_HOSTNAME="${ZITI_CTRL_ADVERTISED_ADDRESS}"
"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_CONTROLLER_HOSTNAME} server certificate ${now}"

Create a new Server cert for the edge api

export ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME="${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}"
export ZITI_EDGE_CONTROLLER_HOSTNAME="${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}"
"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_EDGE_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_EDGE_CONTROLLER_HOSTNAME} server certificate ${now}"

At this point you should be able to follow the rest of those commands at the Update the controller config file location.

I tried it on my quickstart deployed network, but it's hard for me to know if it's exactly right until you try it and verify it.

Let me know how it goes.

kashif · August 20, 2024, 2:15pm

Thank you much again for your help...

Here's the output (ZTN LAN 192.168.1.32)

echo "DNS ENTRIES  : $pki_allow_list_dns"
DNS ENTRIES  : 3,localhost,ztn,ztn,ztn:8443
-
echo "IP ENTRIES   : $pki_allow_list_ip"
IP ENTRIES   : 127.0.0.1
-
echo "CTRL KEY FILE: ${ZITI_PKI_CTRL_KEY}"
CTRL KEY FILE: /root/.ziti/quickstart/ztn/pki/ztn-intermediate/keys/ztn-server.key
-
echo "EDGE KEY FILE: ${ZITI_PKI_EDGE_KEY}"
EDGE KEY FILE: /root/.ziti/quickstart/ztn/pki/ztn-edge-controller-intermediate/keys/ztn.com-server.key

Create a new Server cert for the control plane

Using CA name:  ztn-intermediate
Success

Create a new Server cert for the edge api

Using CA name:  ztn-edge-controller-intermediate
Success

find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'
/root/.ziti/quickstart/ztn/pki/ztn-intermediate/certs/ztn-server.2024-08-20-1837.chain.pem
/root/.ziti/quickstart/ztn/pki/ztn-edge-controller-intermediate/certs/ztn.com-server.2024-08-20-1837.chain.pem

find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_EDGE_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'
/root/.ziti/quickstart/ztn/pki/ztn-edge-controller-intermediate/certs/ztn.com-server.2024-08-20-1837.chain.pem

Added 2 files path in /root/.ziti/quickstart/ztn/ztn.yaml

server_cert: "/root/.ziti/quickstart/ztn/pki/ztn-intermediate/certs/ztn-server.2024-08-20-1837.chain.pem"

server_cert: "/root/.ziti/quickstart/ztn/pki/ztn-edge-controller-intermediate/certs/ztn.com-server.2024-08-20-1837.chain.pem"

Everything goes well but in status I see this TLS error

● ziti-controller.service - Ziti-Controller
     Loaded: loaded (/etc/systemd/system/ziti-controller.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-08-20 14:08:20 UTC; 5min ago
   Main PID: 2586 (ziti)
      Tasks: 8 (limit: 6969)
     Memory: 65.8M
        CPU: 10.088s
     CGroup: /system.slice/ziti-controller.service
             └─2586 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti controller run /root/.ziti/quickstart/ztn/ztn.yaml

Aug 20 14:13:20 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:20.328Z"}
Aug 20 14:13:22 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:22.343Z"}
Aug 20 14:13:23 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"remote error: tls: expired certificate","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13>
Aug 20 14:13:25 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:25.365Z"}
Aug 20 14:13:27 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:27.513Z"}
Aug 20 14:13:28 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"tls: received record with version 301 when expecting version 303","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed>
Aug 20 14:13:30 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"read tcp 192.168.17.32:8441-\u003e82.137.244.207:57070: read: connection reset by peer","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error",">
Aug 20 14:13:32 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:32.615Z"}
Aug 20 14:13:33 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"tls: received record with version 301 when expecting version 303","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed>
Aug 20 14:13:35 ztn ziti[2586]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-20T14:13:35.441Z"}
lines

root@ztn:/home/user# systemctl status ziti-router.service
● ziti-router.service - Ziti-Router for ztn-edge-router
     Loaded: loaded (/etc/systemd/system/ziti-router.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-08-20 14:08:24 UTC; 6min ago
   Main PID: 2598 (ziti)
      Tasks: 8 (limit: 6969)
     Memory: 25.7M
        CPU: 2.462s
     CGroup: /system.slice/ziti-router.service
             └─2598 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti router run /root/.ziti/quickstart/ztn/ztn-edge-router.yaml

Aug 20 14:12:01 ztn ziti[2598]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{kqZ8}","chSeq":121,"connId":116,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-20T14:12:01.405Z","token":"ae5025b7-3634-4d13-ab94-a8422d06001d","type":"EdgeConnectType"}
Aug 20 14:12:01 ztn ziti[2598]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{kqZ8}","chSeq":122,"connId":117,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-20T14:12:01.657Z","token":"ae5025b7-3634-4d13-ab94-a8422d06001d","type":"EdgeConnectType"}
Aug 20 14:12:01 ztn ziti[2598]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{kqZ8}","chSeq":123,"connId":118,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-20T14:12:01.920Z","token":"ae5025b7-3634-4d13-ab94-a8422d06001d","type":"EdgeConnectType"}

TheLumberjack · August 20, 2024, 2:33pm

ugh. for starters, I left a debug output in the commands that was not intentional. the number 3 should not be in your DNS entries... I fixed the command in my post. You might want to redo these commands to remove "3" from your DNS entries sorry about that.

As for the handshake failed, that happens routinely for many reasons. Do those appear constant, or do they appear sporadic? My guess is they are sporadic, if that's the case it's just random people, bots, etc hitting the controller but this also can come from changing the PKI (as you've done here).

Are your tunnelers/sdk apps all working properly? Do things appear to be working fine? If so, I'd guess those are just random people/bots hitting the controller's api. If they are constant and regular (every 10 seconds) then it's likely to be a tunneler. My guess is that it's just random noise but please confirm your tunnelers seem to be working again?

kashif · August 21, 2024, 7:08am

Hi,
Thank you so much, and there is no need for "sorry"

I have redo all the steps

We are using just tunnelers (desktop clients)


echo "DNS ENTRIES  : $pki_allow_list_dns"
DNS ENTRIES  : localhost,ztn,ztn,ztn:8443
echo "IP ENTRIES   : $pki_allow_list_ip"
IP ENTRIES   : 127.0.0.1
echo "CTRL KEY FILE: ${ZITI_PKI_CTRL_KEY}"
CTRL KEY FILE: /root/.ziti/quickstart/ztn/pki/ztn-intermediate/keys/ztn-server.key
echo "EDGE KEY FILE: ${ZITI_PKI_EDGE_KEY}"
EDGE KEY FILE: /root/.ziti/quickstart/ztn/pki/ztn-edge-controller-intermediate/keys/ztn.com-server.key

I am now seeing list of services in ziti desktop client but not able to access it.

Output

systemctl status ziti-controller.service
● ziti-controller.service - Ziti-Controller
     Loaded: loaded (/etc/systemd/system/ziti-controller.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-08-21 06:50:28 UTC; 10min ago
   Main PID: 1857 (ziti)
      Tasks: 9 (limit: 6969)
     Memory: 84.7M
        CPU: 1min 16.279s
     CGroup: /system.slice/ziti-controller.service
             └─1857 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti controller run /root/.ziti/quickstart/ztn/ztn.yaml

Aug 21 07:00:36 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:36.242Z"}
Aug 21 07:00:37 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:37.803Z"}
Aug 21 07:00:37 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:37.874Z"}
Aug 21 07:00:38 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:38.327Z"}
Aug 21 07:00:38 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:38.694Z"}
Aug 21 07:00:39 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:39.330Z"}
Aug 21 07:00:39 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:39.422Z"}
Aug 21 07:00:39 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:39.635Z"}
Aug 21 07:00:40 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"remote error: tls: error decrypting message","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:40.058Z"}
Aug 21 07:00:41 ztn ziti[1857]: {"_context":"tls:0.0.0.0:8441","error":"read tcp 192.168.17.32:8441-\u003e82.137.244.207:54328: read: connection reset by peer","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:00:41.279Z"}
root@ztn:/home/user#

systemctl status ziti-router.service
● ziti-router.service - Ziti-Router for ztn-edge-router
     Loaded: loaded (/etc/systemd/system/ziti-router.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-08-21 06:51:34 UTC; 11min ago
   Main PID: 2447 (ziti)
      Tasks: 8 (limit: 6969)
     Memory: 32.7M
        CPU: 3.440s
     CGroup: /system.slice/ziti-router.service
             └─2447 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti router run /root/.ziti/quickstart/ztn/ztn-edge-router.yaml

Aug 21 07:02:50 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":281,"connId":1078,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:02:50.701Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:02:51 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":282,"connId":1079,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:02:51.214Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:02:52 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":283,"connId":1080,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:02:52.456Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:02:52 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":284,"connId":1081,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:02:52.525Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:02:53 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":285,"connId":1082,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:02:53.095Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:03:08 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":286,"connId":1083,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:03:08.217Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:03:08 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":287,"connId":1084,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:03:08.618Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:03:08 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":288,"connId":1085,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:03:08.961Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:03:09 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":289,"connId":1086,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:03:09.184Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
Aug 21 07:03:09 ztn ziti[2447]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{zArP}","chSeq":290,"connId":1087,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-08-21T07:03:09.726Z","token":"f379b854-c6d1-4f68-8321-94b809b57d32","type":"EdgeConnectType"}
root@ztn:/home/user#

Service log from client

[2024-08-21T07:03:49.681Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:49.941Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.36/WK1pBSyJ/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:49.941Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:50.188Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.37/fxYEhtPI/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:50.188Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:50.448Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.38/Owed-wOA/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:50.448Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:50.706Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.39/9bsmrQZY/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:50.706Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:50.953Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.40/GNnwcyz6/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:50.953Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:51.215Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.41/aH8T-f3s/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:51.215Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:51.459Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.42/W-_G96tG/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:51.459Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:51.719Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.43/yyr-EOce/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:51.719Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-08-21T07:03:51.976Z]   ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.44/GvH08iNr/Connecting] failed to connect, reason=service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators
[2024-08-21T07:03:51.976Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed

And I don't see terminator for the service in ZAC

kashif · August 21, 2024, 7:41am

UPDATE:

The tunneler installed in Ubuntu 18 (web server) needs to be restarted.

Thank you so much. Working now.

But is there any other issues with these errors?

systemctl status ziti-controller.service

ug 21 07:44:15 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"read tcp 192.168.17.32:8441-\u003e181.214.131.108:22957: read: connection reset by peer","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:15.491Z"}
Aug 21 07:44:15 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:15.571Z"}
Aug 21 07:44:16 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:16.495Z"}
Aug 21 07:44:16 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:16.907Z"}
Aug 21 07:44:16 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"tls: received record with version 301 when expecting version 303","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:16.964Z"}
Aug 21 07:44:17 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:17.345Z"}
Aug 21 07:44:18 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:18.253Z"}
Aug 21 07:44:18 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:18.430Z"}
Aug 21 07:44:19 ztn ziti[1525]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:44:19.317Z"}

systemctl status ziti-router.service

Aug 21 07:39:16 ztn ziti[1543]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{eoBZ}","file":"github.com/openziti/channel/v2@v2.0.91/impl.go:323","func":"github.com/openziti/channel/v2.(*channelImpl).rxer","level":"error","msg":"rx error (read tcp 192.168.17.32:8442-\u003e188.229.188.109:25217: read: connection timed out)","time":"2024-08-21T07:39:16.645Z"}
Aug 21 07:40:40 ztn ziti[1543]: {"_context":"tls:0.0.0.0:8442","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.99/tls/listener.go:204","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","time":"2024-08-21T07:40:40.830Z"}
Aug 21 07:42:03 ztn ziti[1543]: {"_context":"{c/e6XgTVYao|@/kNqa}\u003cTerminator\u003e","circuitId":"e6XgTVYao","error":"cannot forward payload, no destination for circuit=e6XgTVYao src=kNqa dst=3VzN","file":"github.com/openziti/fabric@v0.24.2/router/handler_xgress/receive.go:37","func":"github.com/openziti/fabric/router/handler_xgress.(*receiveHandler).HandleXgressReceive","level":"error","msg":"unable to forward payload","origin":1,"seq":5,"time":"2024-08-21T07:42:03.297Z"}
Aug 21 07:42:08 ztn ziti[1543]: {"circuitCount":1,"ctrlId":"ztn","file":"github.com/openziti/fabric@v0.24.2/router/forwarder/faulter.go:102","func":"github.com/openziti/fabric/router/forwarder.(*Faulter).run","level":"warning","msg":"reported forwarding faults","time":"2024-08-21T07:42:08.096Z"}
Aug 21 07:42:10 ztn ziti[1543]: {"_context":"{c/S6XOTVYab|@/9a7x}\u003cTerminator\u003e","circuitId":"S6XOTVYab","error":"cannot forward payload, no destination for circuit=S6XOTVYab src=9a7x dst=YwvG","file":"github.com/openziti/fabric@v0.24.2/router/handler_xgress/receive.go:37","func":"github.com/openziti/fabric/router/handler_xgress.(*receiveHandler).HandleXgressReceive","level":"error","msg":"unable to forward payload","origin":1,"seq":11,"time":"2024-08-21T07:42:10.479Z"}
Aug 21 07:42:15 ztn ziti[1543]: {"error":"no api session found for token [14b6d8f1-3b71-4193-832d-b743b391e497], fingerprints: [map[36ea5f3be2193cfb815915a5c6394db7d92fb16c:0xc000720000]], subjects [[CN=9NgQkMyALY,O=OpenZiti,ST=NY,C=US]]","file":"github.com/openziti/channel/v2@v2.0.91/impl.go:129","func":"github.com/openziti/channel/v2.AcceptNextChannel.func1","level":"error","msg":"failure accepting channel edge with underlay u{classic}-\u003ei{AR8e}","time":"2024-08-21T07:42:15.867Z"}
Aug 21 07:42:23 ztn ziti[1543]: {"circuitCount":1,"ctrlId":"ztn","file":"github.com/openziti/fabric@v0.24.2/router/forwarder/faulter.go:102","func":"github.com/openziti/fabric/router/forwarder.(*Faulter).run","level":"warning","msg":"reported forwarding faults","time":"2024-08-21T07:42:23.097Z"}
Aug 21 07:42:30 ztn ziti[1543]: {"error":"no api session found for token [14b6d8f1-3b71-4193-832d-b743b391e497], fingerprints: [map[36ea5f3be2193cfb815915a5c6394db7d92fb16c:0xc000720b00]], subjects [[CN=9NgQkMyALY,O=OpenZiti,ST=NY,C=US]]","file":"github.com/openziti/channel/v2@v2.0.91/impl.go:129","func":"github.com/openziti/channel/v2.AcceptNextChannel.func1","level":"error","msg":"failure accepting channel edge with underlay u{classic}-\u003ei{Ay2Z}","time":"2024-08-21T07:42:30.930Z"}
Aug 21 07:44:20 ztn ziti[1543]: {"_context":"{c/aD6AhDBvo|@/Y4D6}\u003cTerminator\u003e","error":"payload buffer closed","file":"github.com/openziti/fabric@v0.24.2/router/xgress/xgress.go:564","func":"github.com/openziti/fabric/router/xgress.(*Xgress).forwardPayload","level":"error","msg":"failure to buffer payload","time":"2024-08-21T07:44:20.155Z"}
Aug 21 07:44:25 ztn ziti[1543]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Z5LA}","file":"github.com/openziti/channel/v2@v2.0.91/impl.go:323","func":"github.com/openziti/channel/v2.(*channelImpl).rxer","level":"error","msg":"rx error (read tcp 192.168.17.32:8442-\u003e188.229.188.109:21673: read: connection reset by peer)","time":"2024-08-21T07:44:25.442Z"}
root@ztn:/home/user#

Topic		Replies	Views
Certificate Expired General Questions	19	761	October 27, 2023
Problem with edge router after certificates update Support	4	237	June 18, 2023
Invalid ssl using ssl-passthrough in ziti controller Ziti Overlay	24	84	January 14, 2025
Trying to use router run --extend for the first time	5	204	September 14, 2023
Extremely high resource usage General Questions	5	276	June 14, 2022

Certificate renewal showing path //keys/-server.key

Setup some variables for the server

Create a new Server cert for the control plane

Create a new Server cert for the edge api

Related topics