I spent a fair bit of time today trying to understand all these projects and figure out how they come together to work. There's a bit more going on in there than I feel like I have time to understand fully. I tried to follow the README.md but I wasn't able to successfully understand it enough to feel like i had tested it properly.
Could you give me some more context or show me how to demo/use this?
It might be easiest if you could show me a fully working example without openziti. That would allow me to understand the project better. Is that easy enough to do? If not, I'll keep coming back to this as time permits me to.
Hello,
There are 2 docker containers that are most relevant. publisher-app and subscriber-app. Both connect to livekit(trough ziti). publisher app publishes a video stream( with a red triangle in it(He keeps sending the same image.RGBA 30 times a second). subscriber receives that video feed. the problem here, is that once ziti tunneler is turned off on host, publisher can't publish his video stream.
I will prepare a non-ziti version of this and push to the same repo(different branch) tomorrow.
I found the problem. It was related to backends setup of livekit(turn server couldn't communicate with rtc over ziti). I solved it by setting livekits containers ipv4 address to the same address as the openziti livekits service address.
2 Likes
Oh that's great! I'd love to try it all out and see what you did if I could? Are you going to update your github project so me and others could try it out? I'd also be interested in exploring it on a Ziti TV with you (or not if you prefer not).
Nice going!
I just pushed the code to github, same repo.
Sure, i wont participate, but i can answer any questions if needed.
If you actually want to see the red triangle being streamed, you can install GitHub - livekit-examples/meet: Open source video conferencing app built on LiveKit Components, LiveKit Cloud, and Next.js.(point to wss://12.12.12.12:7880 or wss://livekit.ziti.example:7880), turn on tunnel on host and join testroom.
Awesome. I'll plan to go through all this live on Ziti TV this Friday at 11 am ET then, if that sounds ok with you. If (when?) I hit any bumps, I'll just ask "the chat" and you can help me out? 
Thanks for the Ziti TV idea this'll be fun
EDIT: I guess Friday == tomorrow lol
Sounds good! Will this be a livestream on youtube?
Yes it will. I'm in the midst of setting all that up right now and will follow up later when I get the link ready etc.
I always post it to here, reddit and twitter (and probably on bsky now) when a Ziti TV is scheduled so you can always keep an eye out for them when they are scheduled in any of those places.
1 Like
I've been trying to see "the red triangle" without OpenZiti for starters. I can connect to my http://localhost:3000/rooms/testroom and I can see myself AND i see the publisher, but I'm not seeing the red triangle. The publisher app, after a few moments, indicates it cannot connect to the livekit server:
2024-12-12T09:19:05.653-0500 INFO Connected to Livekit
2024-12-12T09:19:05.684-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:256 Using ICE servers {"servers": [{"urls":["stun:global.stun.twilio.com:3478","stun:stun.l.google.com:19302","stun:stun1.l.google.com:19302"]}]}
2024-12-12T09:19:05.839-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:406 starting to negotiate
...
...
2024-12-12T09:19:20.690-0500 INFO could not connect after timeout
In order to do the Ziti TV, I need to demonstrate the solution without OpenZiti first. Might you be able to help set me straight so I can get that red triangle?
Thanks
Can you post logs that are ...?
sure. there's not a lot to them as it seems to never connect. My flow is:
- start a livekit server:
livekit-server --dev
- start the livekit meet:
pnpm dev (not very relevant imo, but including the info anyway)
- start the publisher:
go run main.go (from ziti-livekit-example/publisher)
cd@192.168.253.239:sg4: ~/git/github/CarlosHleb/ziti-livekit-example/publisher
$ go run main.go
2024-12-12T11:54:47.186-0500 INFO Connected to Livekit
2024-12-12T11:54:47.261-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:256 Using ICE servers {"servers": [{"urls":["stun:global.stun.twilio.com:3478","stun:stun.l.google.com:19302","stun:stun1.l.google.com:19302"]}]}
2024-12-12T11:54:47.411-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:406 starting to negotiate
2024-12-12T11:54:47.411-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:408 create offer {"offer": "v=0\r\no=- 3577768109124250876 1734022487 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 00:19:BD:0B:0B:3F:05:83:D2:E0:71:91:B5:49:EB:EA:C8:7B:F7:E5:36:09:13:51:37:8A:C1:96:1C:5A:25:A8\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:TYwSujNKweYJvRYX\r\na=ice-pwd:BpxNNQwhtGAkGKzOyopBYKZcXZsxcFSp\r\n"}
2024-12-12T11:54:47.420-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:366 successfully set publisher answer
2024-12-12T11:54:47.422-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:151635215 1 udp 2130706431 192.168.253.239 7882 typ host"}
2024-12-12T11:54:47.422-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:233762139 1 udp 2130706431 172.17.0.1 7882 typ host"}
2024-12-12T11:54:47.422-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3036329199 1 tcp 1671430143 192.168.253.239 7881 typ host tcptype passive"}
2024-12-12T11:54:47.646-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:2954730683 1 tcp 1671430143 172.17.0.1 7881 typ host tcptype passive"}
2024-12-12T11:54:47.646-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54617 typ srflx raddr 0.0.0.0 rport 55997"}
2024-12-12T11:54:47.646-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54652 typ srflx raddr 0.0.0.0 rport 52584"}
2024-12-12T11:54:47.647-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54566 typ srflx raddr 0.0.0.0 rport 43732"}
2024-12-12T11:55:03.324-0500 INFO could not connect after timeout
2024-12-12T11:55:03.325-0500 INFO Connected to Livekit
2024-12-12T11:55:03.330-0500 INFO ziti-livekit v2@v2.2.1/engine.go:664 server initiated leave {"reason": "DUPLICATE_IDENTITY", "canReconnect": false}
2024-12-12T11:55:03.331-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:256 Using ICE servers {"servers": [{"urls":["stun:global.stun.twilio.com:3478","stun:stun.l.google.com:19302","stun:stun1.l.google.com:19302"]}]}
2024-12-12T11:55:03.482-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:406 starting to negotiate
2024-12-12T11:55:03.483-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:408 create offer {"offer": "v=0\r\no=- 2701856090875183028 1734022503 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 1C:C0:9D:78:7C:A9:5B:24:55:61:9C:F2:09:C3:A3:26:60:EA:D9:F8:E9:AB:B0:55:84:1B:FD:83:F4:4E:67:22\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:mEmQuhPlkYBdOsTJ\r\na=ice-pwd:shRPAmUkAUeVZsVHSpyRaQHluPUQGwTj\r\n"}
2024-12-12T11:55:03.485-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:366 successfully set publisher answer
2024-12-12T11:55:03.485-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:151635215 1 udp 2130706431 192.168.253.239 7882 typ host"}
2024-12-12T11:55:03.485-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:233762139 1 udp 2130706431 172.17.0.1 7882 typ host"}
2024-12-12T11:55:03.485-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3036329199 1 tcp 1671430143 192.168.253.239 7881 typ host tcptype passive"}
2024-12-12T11:55:03.565-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:2954730683 1 tcp 1671430143 172.17.0.1 7881 typ host tcptype passive"}
2024-12-12T11:55:03.572-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54602 typ srflx raddr 0.0.0.0 rport 51778"}
2024-12-12T11:55:03.574-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54633 typ srflx raddr 0.0.0.0 rport 58821"}
2024-12-12T11:55:03.574-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54627 typ srflx raddr 0.0.0.0 rport 59135"}
2024-12-12T11:55:18.331-0500 INFO could not connect after timeout
2024-12-12T11:55:18.332-0500 INFO Connected to Livekit
2024-12-12T11:55:18.336-0500 INFO ziti-livekit v2@v2.2.1/engine.go:664 server initiated leave {"reason": "DUPLICATE_IDENTITY", "canReconnect": false}
2024-12-12T11:55:18.337-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:256 Using ICE servers {"servers": [{"urls":["stun:global.stun.twilio.com:3478","stun:stun.l.google.com:19302","stun:stun1.l.google.com:19302"]}]}
2024-12-12T11:55:18.487-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:406 starting to negotiate
2024-12-12T11:55:18.487-0500 DEBUG ziti-livekit v2@v2.2.1/transport.go:408 create offer {"offer": "v=0\r\no=- 995192534875547587 1734022518 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 D9:C2:8F:73:FC:D4:B3:3C:54:C1:39:92:07:87:53:54:FE:A1:BD:A0:5B:06:F3:B4:F3:30:08:75:B9:11:04:2A\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:zAPfVsAVWnrJVsNK\r\na=ice-pwd:pFGCjRzTJptkOrRMWpzrsqjpIXURkThf\r\n"}
2024-12-12T11:55:18.489-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:366 successfully set publisher answer
2024-12-12T11:55:18.489-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:151635215 1 udp 2130706431 192.168.253.239 7882 typ host"}
2024-12-12T11:55:18.489-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:233762139 1 udp 2130706431 172.17.0.1 7882 typ host"}
2024-12-12T11:55:18.489-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3036329199 1 tcp 1671430143 192.168.253.239 7881 typ host tcptype passive"}
2024-12-12T11:55:18.528-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:2954730683 1 tcp 1671430143 172.17.0.1 7881 typ host tcptype passive"}
2024-12-12T11:55:18.531-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54639 typ srflx raddr 0.0.0.0 rport 35399"}
2024-12-12T11:55:18.562-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54594 typ srflx raddr 0.0.0.0 rport 33840"}
2024-12-12T11:55:18.562-0500 DEBUG ziti-livekit v2@v2.2.1/engine.go:371 remote ICE candidate {"target": "PUBLISHER", "candidate": "candidate:3476742631 1 udp 1694498815 67.246.244.61 54644 typ srflx raddr 0.0.0.0 rport 52576"}
checkout https://github.com/CarlosHleb/ziti-livekit-example/blob/master/lib/pion-transport/stdnet/net.go
there is no fallback dialer, it won't work without ziti(I will push a fix for it in 10 minutes)
the example is tested with a turn server(livekits built in, checkout livekit.yaml in the repo)
what I would try: try starting livekit-server and livekit-nginx-server from docker. open ports(check install.sh for which ports to open). livekit rtc and turn live inside the same livekit-server container. add /etc/host records for livekit.ziti.example. in docker compose yaml find where it defines --node-ip, set to servers IP.
the publisher logs should show that you are using livekit turn as ice server(your ip:3478).
pushed the ziti dialer fallback
Thanks. I wasn't able to get it to work successfully and i didn't get any time today to try it again so I'll just fumble around with it on the live stream tomorrow. 
Theres another issue with stability of connection. The publisher connection drops at random times(and reconnects after a couple seconds). This results in freezed videos, before publisher reconnects again. Happens more often on windows, less often on ubuntu.
edge router logs give this:
ziti-edge-router-1 | [ 373.752] ERROR ziti/router/xgress_common.(*XgressConn).WritePayload: {error=[crypto failed]} crypto failed on msg of size=69, headers=map[]
ziti-edge-router-1 | [ 373.752] WARNING ziti/router/xgress.(*Xgress).tx [{c/L0UdlPJDa|@/bX2o}<Terminator>]: {circuitId=[L0UdlPJDa] seq=[20548] origin=[Initiator]} write failed (crypto failed), closing xgress
ziti-edge-router-1 | [ 373.752] WARNING ziti/router/xgress.(*Xgress).rx [{c/L0UdlPJDa|@/bX2o}<Terminator>]: read failed (read udp 12.34.56.6:41132->12.34.56.78:3478: use of closed network connection)
ziti-edge-router-1 | [ 373.752] ERROR ziti/router/handler_xgress.(*receiveHandler).HandleXgressReceive [{c/L0UdlPJDa|@/xnnY}<Initiator>]: {circuitId=[L0UdlPJDa] seq=[20553] error=[cannot forward payload, no destination for circuit=L0UdlPJDa src=xnnY dst=bX2o] origin=[Initiator]} unable to forward payload
ziti-edge-router-1 | [ 375.018] WARNING ziti/router/forwarder.(*Faulter).run: {ctrlId=[ziti-controller] circuitCount=[1]} reported forwarding faults
Any idea what does the crypto fail mean?
Are there any settings in router/ctrl that can couse this?
the publisher with logrus in debug level, give this:
DEBU[0016] close: begin connId=3 marker=leqzTKID
DEBU[0016] received 12 bytes (msg type: 60785) chSeq=96 connId=3 edgeSeq=45 type=EdgeStateClosedType
DEBU[0016] received ConnState_CLOSED message, closing connection connId=3 marker=leqzTKID
DEBU[0016] close: end connId=3 marker=leqzTKID
Hello,
I updated to repo. It now builds livekit from source(easier to debug). It looks like the connection gets dropped by ziti. livekit forwardRTP function listens on a buffer, it stops receives bytes.
Could it be related to failing crypto error from above?
Update: I started the project. publisher started publishing and then i killed ziti-edge-router. same error on publisher side, but not on livekit. So it looks like, livekit to ziti edge stays alive, but publisher to edge router gets dropped by ziti.
What could be the reasons for ziti sending by publisher app:
publisher-app-1 | e[37mDEBUe[0m[0028] received ConnState_CLOSED message, closing connection e[37mconnIde[0m=3 e[37mmarkere[0m=zKgSO6ef
Hi @CarlosHleb,
There are often many different reasons that this can happen. Are there any logs in the controller or router that seem relevant? Is anything changing the identity's authorization? Does this maybe happen after 30 minutes by chance?
Any extra details would help, if there any relevant log messages, please share?
After talking about this more with someone, this sounds like it's probably just the livekit server closing the connection. Maybe the livekit server has a relevant log?
