Missing Terminators?

apinohixcorp · November 12, 2024, 1:58am

I recently configured several services assigned to a vm that I'm using for managing print services. I have four services:

Factory Printers SSH
Factory Printers VNC
Factory Printers DirectPrint
Factory Printers CUPS

Suddenly these stopped working after I created them, with an error that there was no terminator for any of these, only for a few other services that I have. Also I see two terminators for the same service macola-hix-sql, but I'm not sure what that's about:

The missing terminators made my services unreachable. The only solution I could think of was to delete one of them and remake it and see what happens, so I deleted and recreated the SSH service, and it did add it back in, and I was able to SSH into the VM:

I also found just simply deleting the host and intercept config and reattaching them recreated the terminators as well.

My question is, what would have happened that made the terminators disappear and make my service unreachable? I want to make sure that doesn't happen again.

TheLumberjack · November 12, 2024, 12:27pm

I've never seen this type of behavior myself. It makes me think a misconfiguration was applied, invalidating the prior configuration.

And all four services worked after you did this? Recreating each config ended up restoring that one service? Hmmmm. That's a bit strange. There's a chance whatever flow you were using triggered a bug that we would want to identifiy/find/fix.

The only problem is that I'd guess you don't know exactly what sorts of things you were doing to cause the issue.

I don't know if we'll be able to get to the bottom of 'what happened' here but I'd look through the logs on the tunneler side. When a service becomes available you'll see something like this in the logs:


(562644)[       70.252]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:822 on_service() hosting server_address[tcp:localhost:22] service[zsshSvc]
(562644)[       70.252]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1295 on_event() =============== service event (added) - zsshSvc:1bHvjStZKmY2VpZxtFgPzm ===============

Then when it's removed for 'whatever' reason you'll see:

(562644)[      100.329]    INFO tunnel-cbs:ziti_tunnel_cbs.c:568 ziti_sdk_c_on_service() service unavailable: zsshSvc
(562644)[      100.329]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1271 on_event() =============== service event (removed) - zsshSvc:1bHvjStZKmY2VpZxtFgPzm ===============

I'd look through the logs and see if you find those related events and if you do, try to remember what might have happened at that time.

Or, if you discover how to reproduce this issue let us know the steps and we'll try to reproduce it on our end (or let you know why it happened if it's obvious to us)

Cheers

TheLumberjack · November 12, 2024, 12:29pm

I forgot to answer your question... As to WHAT causes it -- a lot of things can cause it (which is why i forgot to answer). In my experience, it's almost always "something I did". Usually it's when I'm modifying configurations, I'll mess up a service policy etc. It's "usually" my fault. Without knowing more though, I don't think I can give you a concrete "this is why" answer.

scareything · November 12, 2024, 3:18pm

You shouldn't need to manually clear/recreate terminators. They are supposed to be created automatically when an identity such as a hosting tunneler binds a service. You inadvertently caused the service to re-bind when you updated the service configurations, btw.

From what you're describing it would be helpful to see logs at TRACE level from the hosting tunneler that was providing the terminator that disappeared. I'd expect to see some kind of connection failure either with the controller or ER that coincides with the terminator removal. The tunneler should recreate its terminator when the connection is re-established, but there may be a bug that prevents this from happening.

apinohixcorp · November 15, 2024, 3:17pm

Sorry for the late reply here, had a busy couple of days.

@TheLumberjack And all four services worked after you did this? Recreating each config ended up restoring that one service? Hmmmm. That's a bit strange. There's a chance whatever flow you were using triggered a bug that we would want to identifiy/find/fix.

No, recreating/reattaching each service recreated the terminator for each service separately.

This was working fine after I recreated the services for a while but suddenly the terminators seemed to have disappeared again:

I tried reattaching the services again, but this did not recreate the terminators this time. I also tried creating a new service from scratch and also did not get a new terminator.

I checked the logs on the ziti router host:

Nov 15 14:30:24 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":562,"connId":225,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:30:24.356Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:30:24 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":563,"connId":226,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:30:24.940Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:30:25 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":564,"connId":227,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:30:25.530Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:30:26 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":565,"connId":228,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:30:26.120Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:30:26 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":566,"connId":229,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:30:26.695Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:44:48 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":581,"connId":230,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:44:48.037Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:44:48 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":582,"connId":231,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:44:48.622Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:44:49 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":583,"connId":232,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:44:49.197Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:44:49 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":584,"connId":233,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:44:49.777Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:44:50 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":585,"connId":234,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:44:50.375Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:50:19 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":592,"connId":235,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:50:19.035Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:50:19 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":593,"connId":236,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:50:19.625Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:50:20 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":594,"connId":237,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:50:20.214Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:50:20 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":595,"connId":238,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:50:20.794Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}
Nov 15 14:50:21 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DzM6}","chSeq":596,"connId":239,"edgeSeq":0,"error":"service 2amXBVkTxUGOfdmPnHC2Kv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-11-15T14:50:21.378Z","token":"cee2214f-8dc9-4eff-a508-632b2cfbe0a2","type":"EdgeConnectType"}

I tried restarting the ziti-edge-tunneler (sudo systemctl restart ziti-edge-tunneler.service) on the target VM and got this log output on the router host.

Nov 15 14:54:14 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DLqm}","bindConnId":1,"chSeq":2,"connId":1,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:539","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":";\ufffdµ\ufffd\ufffd~U\u0010|A\ufffd\ufffd\ufffd\ufffd\u0001|\ufffd\ufffd\ufffd\u003en#\ufffd\ufffdiX\ufffd\ufffd\u001elY","msg":"establishing terminator","routerId":"DZ-TsGYCIf","sessionToken":"d1471ee5-e118-4e74-8e72-1b9770727388","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:14.296Z","type":"EdgeBindType"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:162","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue","level":"info","msg":"queuing terminator to send create","state":1,"terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:14.296Z","token":"d1471ee5-e118-4e74-8e72-1b9770727388"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:570","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator","level":"info","msg":"sending create terminator v2 request","routerId":"DZ-TsGYCIf","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:14.296Z","token":"d1471ee5-e118-4e74-8e72-1b9770727388"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:150","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).updateState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"create notification received","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:14.303Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"connId":1,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:974","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":7390195,"msg":"terminator established","routerId":"DZ-TsGYCIf","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:14.303Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DLqm}","bindConnId":2,"chSeq":3,"connId":2,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:539","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":"3}}ǆ\ufffd\ufffd\u001eצ\u0013P\ufffd|1\ufffd\u0018(\ufffdw˫\ufffd~\"\ufffdr'\f\ufffdӝ","msg":"establishing terminator","routerId":"DZ-TsGYCIf","sessionToken":"be570be3-b281-4c8a-a403-1363aa8e4d4d","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:14.393Z","type":"EdgeBindType"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:162","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue","level":"info","msg":"queuing terminator to send create","state":1,"terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:14.393Z","token":"be570be3-b281-4c8a-a403-1363aa8e4d4d"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:570","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator","level":"info","msg":"sending create terminator v2 request","routerId":"DZ-TsGYCIf","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:14.393Z","token":"be570be3-b281-4c8a-a403-1363aa8e4d4d"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:150","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).updateState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"create notification received","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:14.400Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"connId":2,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:974","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":6714281,"msg":"terminator established","routerId":"DZ-TsGYCIf","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:14.400Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DLqm}","bindConnId":3,"chSeq":4,"connId":3,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:539","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":"\ufffd\ufffd\b^\ufffdƓ-e\ufffd\u001a\ufffd\ufffdC\ufffdY\ufffd\ufffd5\u003c\ufffd\ufffdU\u0018\ufffd\u001c\ufffd\ufffd4\u001b\ufffdC","msg":"establishing terminator","routerId":"DZ-TsGYCIf","sessionToken":"b1199fc3-aae3-4a9e-b669-30bdec572d6d","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:14.489Z","type":"EdgeBindType"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:162","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue","level":"info","msg":"queuing terminator to send create","state":1,"terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:14.489Z","token":"b1199fc3-aae3-4a9e-b669-30bdec572d6d"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:570","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator","level":"info","msg":"sending create terminator v2 request","routerId":"DZ-TsGYCIf","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:14.489Z","token":"b1199fc3-aae3-4a9e-b669-30bdec572d6d"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:150","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).updateState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"create notification received","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:14.496Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"connId":3,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:974","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":7220507,"msg":"terminator established","routerId":"DZ-TsGYCIf","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:14.496Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DLqm}","bindConnId":4,"chSeq":5,"connId":4,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:539","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":"Gr\ufffd\u000f1\u001aW\ufffd\ufffd\ufffdnU\ufffd\u0012\ufffd֎\ufffd\ufffd\ufffd\u001c\r#\ufffd\u0019\ufffd)\ufffd!\ufffd\\n","msg":"establishing terminator","routerId":"DZ-TsGYCIf","sessionToken":"ad3f5b68-70e2-43bd-8e5e-fff7212700d3","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:14.585Z","type":"EdgeBindType"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:162","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue","level":"info","msg":"queuing terminator to send create","state":1,"terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:14.585Z","token":"ad3f5b68-70e2-43bd-8e5e-fff7212700d3"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:570","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator","level":"info","msg":"sending create terminator v2 request","routerId":"DZ-TsGYCIf","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:14.585Z","token":"ad3f5b68-70e2-43bd-8e5e-fff7212700d3"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:150","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).updateState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"create notification received","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:14.591Z"}
Nov 15 14:54:14 hix-ziti ziti[25450]: {"connId":4,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:974","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":6232644,"msg":"terminator established","routerId":"DZ-TsGYCIf","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:14.591Z"}
Nov 15 14:54:15 hix-ziti ziti[25450]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{DLqm}","bindConnId":0,"chSeq":6,"connId":0,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:539","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":"M~\ufffd(\ufffd\ufffd-\ufffd\ufffd`;\u0003u\ufffd/\ufffd\u0005\ufffd\ufffd\ufffd\u00262\ufffdvq\ufffd)\ufffd\ufffd\u0017\t\ufffd","msg":"establishing terminator","routerId":"DZ-TsGYCIf","sessionToken":"6f785848-d4ca-47d1-a91d-283af03c8717","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:15.025Z","type":"EdgeBindType"}
Nov 15 14:54:15 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:162","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue","level":"info","msg":"queuing terminator to send create","state":1,"terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:15.026Z","token":"6f785848-d4ca-47d1-a91d-283af03c8717"}
Nov 15 14:54:15 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:570","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator","level":"info","msg":"sending create terminator v2 request","routerId":"DZ-TsGYCIf","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:15.026Z","token":"6f785848-d4ca-47d1-a91d-283af03c8717"}
Nov 15 14:54:15 hix-ziti ziti[25450]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:150","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).updateState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"create notification received","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:15.032Z"}
Nov 15 14:54:15 hix-ziti ziti[25450]: {"connId":0,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:974","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":6386324,"msg":"terminator established","routerId":"DZ-TsGYCIf","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:15.032Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{XxRW}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:25.037Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{XxRW}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:25.037Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"connId":1,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:972","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":10741421501,"msg":"received additional terminator created notification","routerId":"DZ-TsGYCIf","terminatorId":"6NxYau4uFhivE59YX9icBm","time":"2024-11-15T14:54:25.037Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"connId":2,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:972","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":10643844148,"msg":"received additional terminator created notification","routerId":"DZ-TsGYCIf","terminatorId":"2ebg7R2Ivhz6Mn3ELG8pcY","time":"2024-11-15T14:54:25.037Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{XxRW}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:25.082Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{XxRW}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:25.082Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"connId":3,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:972","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":10593333061,"msg":"received additional terminator created notification","routerId":"DZ-TsGYCIf","terminatorId":"3FToz46NoGwTSQQUKLuKDV","time":"2024-11-15T14:54:25.082Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"connId":4,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:972","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":10497077544,"msg":"received additional terminator created notification","routerId":"DZ-TsGYCIf","terminatorId":"4kIp5pjcbjgQXcYD33Ekhu","time":"2024-11-15T14:54:25.082Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{XxRW}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:25.126Z"}
Nov 15 14:54:25 hix-ziti ziti[25450]: {"connId":0,"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:972","func":"github.com/openziti/ziti/router/xgress_edge.(*markEstablishedEvent).handle","level":"info","lifetime":10101080387,"msg":"received additional terminator created notification","routerId":"DZ-TsGYCIf","terminatorId":"2bmctNBLXytGPz9aA6jV4V","time":"2024-11-15T14:54:25.126Z"}

Restarting the edge tunneler recreated all of the services including the new one (Factory Printers SSH 2)

I was then able to ssh using the ziti network again on all four services, but I'm not quite sure I understand why these keep getting deleted.

I did make all of these service names human readable with spaces just out of preference, but it seems like these are the only ones that seem to disappear after a few days and the other two that use no spaces (macola-hix-sql and order-management) don't disappear. That being said macola-hix-sql and order-management both have the ziti windows edge tunneler as they are windows servers, while the factory printers services all have the debian ziti edge tunneler installed, so it could be just OS dependent and not service names. I'm really lost at this point.

scareything · November 15, 2024, 3:48pm

This is certainly an unusual problem. Could you please set the log level to TRACE on the Ubuntu ziti-egde-tunnel and share the logs the next time you notice the terminator disappears? Let me know if you're not sure how to change the log level... If you installed it from apt and you're starting it with systemctl then you can edit the log level in the ziti-edge-tunnel.env file followed by systemctl daemon-reload and restarting ziti-edge-tunnel.

It would also be helpful to see the edge router logs that cover the timespan of the terminators disappearing.

Also can you tell me which version of ziti-edge-tunnel and the openziti controller/router you're running?

Thanks

apinohixcorp · November 15, 2024, 4:49pm

I checked the logs on the vm running the tunnel:

hixadmin@hixfactoryprintersvm:~$ sudo journalctl -f -u ziti-edge-tunnel.service
[sudo] password for hixadmin:
-- Journal begins at Tue 2024-10-29 17:51:55 CDT. --
Nov 15 08:54:13 hixfactoryprintersvm systemd[1]: Starting Ziti Edge Tunnel...
Nov 15 08:54:13 hixfactoryprintersvm ziti-edge-tunnel.sh[2763185]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Nov 15 08:54:13 hixfactoryprintersvm systemd[1]: Started Ziti Edge Tunnel.
Nov 15 08:54:13 hixfactoryprintersvm ziti-edge-tunnel[2763187]: About to run tunnel service... ziti-edge-tunnel
Nov 15 08:54:13 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.252]    WARN ziti-sdk:ziti.c:548 ext_jwt_singers_cb() ztx[0] failed to get external auth providers: The request could not be completed. The session is not authorized or the credentials are invalid
Nov 15 08:54:14 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.519]    WARN ziti-edge-tunnel:resolvers.c:399 try_libsystemd_resolver() libsystemd resolver unsuccessful. Falling back to legacy resolvers
Nov 15 08:54:14 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.519]    WARN ziti-edge-tunnel:tun.c:277 find_dns_updater() Adding ziti resolver to /etc/resolv.conf. Ziti DNS functionality may be impaired
Nov 15 08:54:14 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.519]    WARN ziti-edge-tunnel:resolvers.c:478 make_copy() could not create copy[/etc/resolv.conf.bkp]: permission denied
Nov 15 08:54:14 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.519]   ERROR ziti-edge-tunnel:resolvers.c:523 dns_update_etc_resolv() cannot open /etc/resolv.conf: Permission denied
Nov 15 08:54:14 hixfactoryprintersvm ziti-edge-tunnel[2763187]: (2763187)[        0.519]    WARN ziti-edge-tunnel:resolvers.c:524 dns_update_etc_resolv() run as 'root' or manually update your resolver configuration. Ziti DNS must be the first resolver: 100.64.0.2

It seems like there might be a permissions issue with the vm regarding resolv.conf. I found the instruction at Tunneling on Debian GNU/Linux | OpenZiti which specifies some instructions regarding systemd-resolved.

I enabled systemd-resolved

sudo systemctl enable --now systemd-resolved

Is this what is meant by delegate management of /etc/resolv.conf to systemd-resolved?

sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

apinohixcorp · November 15, 2024, 5:17pm

Here's the logs from the time when I recreated the services after the terminators were missing the first time:
ziti-tunneler-logs.txt (274.6 KB)

And here are the logs from the original time period when I first installed and started the tunneler until the first time I lost the service:
ziti-tunneler-log-first.txt (4.5 MB)

It's pretty clear that it wasn't able to resolve the DNS name, but it shocks me that I was able to connect to the service at all in the first place? Appreciate your patience. I did enable the systemd-resolved and ran the permissions delegation change and it looks like I'm no longer getting those DNS failures. So maybe that's all I missed in the first place?

scareything · November 15, 2024, 9:08pm

I'm still not sure what exactly what you're using the ziti-edge-tunnel on Ubuntu for. I was assuming that you were only using for the "server" side of the connection - e.g. the last mile between the Ziti network and the print server (or whatever is supposed to be listening on port 631). The resolv.conf warnings would only be relevant if you are intercepting connections with this ziti-edge-tunnel, and even then it would only matter if you have hostnames (vs IP addresses) in your intercept.v1 configuration.

There were a few transitions in the setup so I won't bother with a line-by-line analysis. At the end of the logs I see this:

ERROR tunnel-cbs:ziti_hosting.c:258 on_hosted_tcp_server_connect_complete() hosted_service[Factory Printers CUPS], client[ARTUROTHINKPAD] client_src_addr[tcp:100.64.0.1:53903]: connect to tcp:127.0.0.1:631 failed: connection refused

This tells me that:

The terminator existed (at least at this point in time), otherwise you'd not have seen an incoming connection from ARTUROTHINKPAD.
The server that the host.v1 configuration specifies was either not running or not listening at tcp:127.0.0.1:631 when the connections were attempted.

apinohixcorp · November 19, 2024, 6:29pm

I guess I'm not sure I understand how I would get to the target host if I don't have a tunneler installed on that VM? I'm actually running a debian VM with a CUPS print server on port 631. I'm using the ziti edge tunneler on that VM based on these instructions:

Tunneling on Debian GNU/Linux | OpenZiti

My understanding is that if I want to create a service in the ziti network, I need a host with a tunneler (the debian VM) and a client with a tunneler (my windows laptop). Then I can enroll the identities for each and bind the service to the host and dial it with the client and the tunnelers will let me access the host. Here's a diagram of how I have it configured for both SSH access (port22) and the CUPS print service (port 631). Is my understanding wrong?

I was trying a lot of different stuff trying to get to the VM, including regular ssh without the ziti tunneler, so the logs could have a lot of different things going on.

I'm not sure why the resolv.conf warnings were happening then, but my understanding was that the edge tunneler was using the DNS resolution to access the Controller somehow, or at least trying to make the configurations for the services it was hosting. But this Debian 11 VM had legacy resolve instead of systemd-resolved package, so it wasn't able to dynamically edit the resolv.conf file. When I installed sudo apt install systemd-resolved and ran this those warnings stopped and I haven't had an issue with the terminators since.

sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Reinhard · November 19, 2024, 6:41pm

The problem exists only via ubuntu and the ubuntu script

TheLumberjack · November 19, 2024, 6:50pm

The diagram is excellent and it seems to me like you understand things pretty well. I ~~might use~~ will probably use your diagram styling -- i like the simplified approach you took!

Are you still receiving no terminators issues or are you all set now?

qrkourier · November 19, 2024, 6:53pm

Yep, installing systemd-resolved is the recommended approach in the Debian GNU/Linux tunneler instructions linked above (click on the tab "Debian GNU/Linux").

This is recommended because it's more secure than the alternative, running the tunneler as root.

apinohixcorp · November 19, 2024, 7:02pm

Glad to hear it, I thought I was missing something. Here's a link to the Excalidraw version if you want to adapt it yourself!

I'm not receiving any errors after I installed systemd-resolved (the edge tunneler was/is running as user "ziti" by default, which is why it was failing before since it wasn't "root). I think this was effectively the solution, specifically in my case for Debian 11 which didn't have systemd-resolved installed/enabled by default, and resolv.con properly configured.

qrkourier · November 19, 2024, 7:10pm

Thats' right. You've got it.

We're contemplating ways to make it easier on Debian. For now, the best way is to first install systemd-resolved. When I tested this on Bookworm and Trixie, only installing the package systemd-resolved was necessary because it symlinked my unmodified /etc/resolv.conf to systemd-resolved's stub resolver configuration file, which sends all queries to the local nameserver provided by systemd-resolved that is being configured for Ziti via dbus.

apinohixcorp · November 20, 2024, 8:27pm

Just a note that it may be worth adding this instruction to RHEL based distros as well as I found systemd-resolve was not installed in my Rocky Linux 9 VM by default.

$ sudo dnf install systemd-resolved -y
$ sudo systemctl enable --now systemd-resolved

Topic		Replies	Views
Weird connection problems	10	89	January 14, 2025
Creating & Removing Terminator Loop	10	190	November 21, 2023
Missing Terminators after Controller reboot Support	9	40	May 6, 2025
Update Controller - Terminators are terminated Support	3	30	November 28, 2024
Question about terminator General Questions	30	707	August 30, 2023

Missing Terminators?

Related topics