Problem with edge router after certificates update

Support
1

Hello.

After I renewed certificates on controller I have problems with router connections

I’ve updated certificates here:

identity:
  cert:                 "/root/.ziti/quickstart/openziti/pki/openziti-intermediate/certs/openziti-client-2023.cert"
  server_cert:          "/root/.ziti/quickstart/openziti/pki/openziti-intermediate/certs/openziti-server-2023.chain.pem"
  key:                  "/root/.ziti/quickstart/openziti/pki/openziti-intermediate/keys/openziti-server-2023.key"
  ca:                   "/root/.ziti/quickstart/openziti/pki/cas.pem"

and here

web:
  - name: client-management
    identity:
      ca:          "/root/.ziti/quickstart/openziti/pki/localziti-intermediate/certs/localziti-intermediate.cert"
      key:         "/root/.ziti/quickstart/openziti/pki/localziti-intermediate/keys/localziti-server-2023.key"
      server_cert: "/root/.ziti/quickstart/openziti/pki/localziti-intermediate/certs/localziti-server-2023.chain.pem"
      cert:        "/root/.ziti/quickstart/openziti/pki/localziti-intermediate/certs/localziti-client-2023.cert"

router logs:

Jun 16 16:25:28 openziti ziti-router[1264356]: {"file":"github.com/openziti/fabric@v0.22.24/router/router.go:455","func":"github.com/openziti/fabric/router.(*Router).startControlPlane","level":"info","msg":"connecting to controller at endpoing [tls:openziti:6262]","time":"2023-06-16T16:25:28.208Z"}
Jun 16 16:25:29 openziti ziti-router[1264356]: {"error":"error connecting ctrl (context deadline exceeded)","file":"github.com/openziti/ziti/ziti/router/run.go:122","func":"github.com/openziti/ziti/ziti/router.run","level":"fatal","msg":"error starting","time":"2023-06-16T16:25:29.209Z"}
Jun 16 16:25:29 openziti systemd[1]: ziti-router.service: Main process exited, code=exited, status=1/FAILURE
Controller logs 

Jun 16 16:31:10 openziti ziti-controller[1264150]: {"_context":"ch{uo7E2zMTrg}-\u003eu{classic}-\u003ei{dV9Q}","file":"github.com/openziti/fabric@v0.22.24/controller/handler_ctrl/close.go:48","func":"github.com/openziti/fabric/controller/handler_ctrl.(*xctrlCloseHandler).HandleClose","level":"info","msg":"closing Xctrl instances","time":"2023-06-16T16:31:10.505Z"}
Jun 16 16:31:10 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/handler_ctrl/close.go:35","func":"github.com/openziti/fabric/controller/handler_ctrl.(*closeHandler).HandleClose","level":"warning","msg":"disconnected","routerId":"uo7E2zMTrg","time":"2023-06-16T16:31:10.597Z"}
Jun 16 16:31:14 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/api/timeouts.go:126","func":"github.com/openziti/fabric/controller/api.(*timeoutHandler).ServeHTTP","level":"error","method":"GET","msg":"timeout for request hit, returning Service Unavailable 503","time":"2023-06-16T16:31:14.389Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/edge/client/v1/current-api-session/service-updates","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
Jun 16 16:31:14 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/api/timeouts.go:126","func":"github.com/openziti/fabric/controller/api.(*timeoutHandler).ServeHTTP","level":"error","method":"GET","msg":"timeout for request hit, returning Service Unavailable 503","time":"2023-06-16T16:31:14.510Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/edge/client/v1/current-api-session/service-updates","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
Jun 16 16:31:15 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/api/timeouts.go:126","func":"github.com/openziti/fabric/controller/api.(*timeoutHandler).ServeHTTP","level":"error","method":"GET","msg":"timeout for request hit, returning Service Unavailable 503","time":"2023-06-16T16:31:15.190Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/edge/client/v1/current-api-session/service-updates","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"error":"http: Handler timeout","file":"github.com/openziti/fabric@v0.22.24/controller/api/responder.go:124","func":"github.com/openziti/fabric/controller/api.(*ResponderImpl).RespondWithProducer","level":"error","msg":"could not respond, writing to response failed","path":"/edge/client/v1/current-api-session/service-updates","requestId":"JG0A8J05o","time":"2023-06-16T16:31:21.007Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"error":"http: Handler timeout","file":"github.com/openziti/fabric@v0.22.24/controller/api/responder.go:124","func":"github.com/openziti/fabric/controller/api.(*ResponderImpl).RespondWithProducer","level":"error","msg":"could not respond, writing to response failed","path":"/edge/client/v1/current-api-session/service-updates","requestId":"ukzA8D05q","time":"2023-06-16T16:31:21.113Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"error":"http: Handler timeout","file":"github.com/openziti/fabric@v0.22.24/controller/api/responder.go:124","func":"github.com/openziti/fabric/controller/api.(*ResponderImpl).RespondWithProducer","level":"error","msg":"could not respond, writing to response failed","path":"/edge/client/v1/current-api-session/service-updates","requestId":"MmlA8Dl5o","time":"2023-06-16T16:31:21.193Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/network/network.go:815","func":"github.com/openziti/fabric/controller/network.(*Network).Run","level":"info","msg":"changed router","routerId":"uo7E2zMTrg","time":"2023-06-16T16:31:21.098Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/edge@v0.24.125/controller/env/broker.go:116","func":"github.com/openziti/edge/controller/env.(*Broker).RouterDisconnected.func1","level":"info","msg":"broker detected router with id uo7E2zMTrg disconnecting","routerFingerprint":"87404f566d218d803d63dd6cf48fe69e4453bec7","routerId":"uo7E2zMTrg","routerName":"openziti-edge-router","time":"2023-06-16T16:31:21.303Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/edge@v0.24.125/controller/sync_strats/sync_instant.go:181","func":"github.com/openziti/edge/controller/sync_strats.(*InstantStrategy).RouterDisconnected","level":"info","msg":"edge router [uo7E2zMTrg] disconnecting","routerFingerprint":"87404f566d218d803d63dd6cf48fe69e4453bec7","routerId":"uo7E2zMTrg","routerName":"openziti-edge-router","time":"2023-06-16T16:31:21.306Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/handler_ctrl/bind.go:82","func":"github.com/openziti/fabric/controller/handler_ctrl.(*bindHandler).BindChannel","level":"info","msg":"router supports heartbeats","routerId":"uo7E2zMTrg","routerVersion":"v0.27.5","time":"2023-06-16T16:31:21.809Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/handler_ctrl/accept.go:112","func":"github.com/openziti/fabric/controller/handler_ctrl.(*CtrlAccepter).Bind","level":"info","msg":"accepted new router connection [r/uo7E2zMTrg]","routerId":"uo7E2zMTrg","time":"2023-06-16T16:31:21.811Z"}
Jun 16 16:31:21 openziti ziti-controller[1264150]: {"file":"github.com/openziti/edge@v0.24.125/controller/env/broker.go:86","func":"github.com/openziti/edge/controller/env.(*Broker).RouterConnected.func1","level":"error","msg":"duplicate router connection detected [id: uo7E2zMTrg], ignoring","routerFingerprint":"87404f566d218d803d63dd6cf48fe69e4453bec7","routerId":"uo7E2zMTrg","routerName":"openziti-edge-router","time":"2023-06-16T16:31:21.897Z"}
Jun 16 16:31:22 openziti ziti-controller[1264150]: {"file":"github.com/openziti/fabric@v0.22.24/controller/network/network.go:815","func":"github.com/openziti/fabric/controller/network.(*Network).Run","level":"info","msg":"changed router","routerId":"uo7E2zMTrg","time":"2023-06-16T16:31:22.191Z"}
2

Hi @Wild, Sounds like you are saying your certificates expired and you recreated them somehow, and after doing that your routers can’t connect any more? Can you explain the process you used to regenerate the certificates?

I feel like this is probably something we need to write up a guide for in general. We’ll get your network back working, but it will probably require some back and forth here in the forum.

Routers are usually pretty easy to provision. It might end up easier up just delete and recreate your router.

1 Like
3

Thank you for your help.
Yes, routers show tls error (before it was about outdated cert)
I’ve created them with ziti pki create server/client and add 2023 in naming.

If I’ll recreate router with same name all my configs will be functional?

4

Found, that I had to change ~/.ziti/quickstart/$(hostname)/$(hostname).env, not only in yaml file
Everything is working fine now.

1 Like
5

Glad you got it sorted! Sorry I didn’t get a chance to answer your question. Usually, if you use attribute-based policies (not “at-based”, aka directly mentioning the object), just deleting/recreating the router (or identity or service etc) with the proper attributes will “just work” and be functional. If you use “at-based” references, when you remove the object (like a router and router policies) you’ll have to repair the policy too. But – seems like you figured it all out, nice!