Zitifiying webrtc golang - net.ListenUDP?

I am trying to zitify pions/webrtc package. the openzities dialer doesn't recognize function ListenUDP, are there any known workarounds?

	ZitiTransport = http.DefaultTransport.(*http.Transport).Clone() // copy default transport
	ZitiTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
		dialer := ZitiContexts.NewDialerWithFallback(ctx, fallback)
		return dialer.Dial(network, addr)
	}
	ZitiTransport.Dial = func(network, addr string) (net.Conn, error) {
		ctx := context.Background()
		dialer := ZitiContexts.NewDialerWithFallback(ctx, fallback)
		return dialer.Dial(network, addr)
	}
        // ListenUDP function not found
	ZitiTransport.ListenUDP = func(network, addr string) (net.Conn, error) {
		ctx := context.Background()
		dialer := ZitiContexts.NewDialerWithFallback(ctx, fallback)
                // ListenUDP function not found
		return dialer.ListenUDP(network, addr)
	}

Hi @CarlosHleb, I would say that for me, when I look to use OpenZiti for secure communications, the layer4 protocol isn't actually relevant. You'd just "listen" on the overlay, and then a client would "send bytes" to that listener.

Is there a reason you're reaching for a UDP listener?

If you're looking to intercept actual underlay UDP packets and relay them over ziti, I would think that's what a tunneler would be for. So I must admit, I could probably use more information to give a better answer.

I dunno if that helps at all, but it's a start? :slight_smile:

Yes, it works with a tunneler(from browser).

there is a requirement, that the app should run without tunneler. Meaning, compiled binary is installed on end-users device, connects to openziti services using openzitis golang sdk.

Sure, I get that. I would think you would zitify the "server" side using Listen and then from the clients, they would just send bytes to the server. TCP/UDP wouldn't be relevant at that point, I don't believe.

You'd just "listen" for or "receive" bytes, read those bytes and dispatch the bytes accordingly.

I don't think i understand what you mean.

I havent zitified the server. server is livekit behind ziti.
I am trying to zitify end-user app(uses pion/webrtc), because end-user app needs to connect to ziti so it can access the server.

If i understand it correctly, end-user app needs to be able to send and receive udp data. i forked relevant pion repos. currently it gets stuck on ice state connecting from the livekit and end-user app gets stuck at ice state checking.

The relevant debug code:

type zitiPacketConn struct {
	zitiNet net.Conn
	net     net.PacketConn
	network string
}

func (z *zitiPacketConn) ReadFrom(b []byte) (int, net.Addr, error) {
	// Read data from the Ziti connection
	var t []byte
	s, err := z.zitiNet.Read(t)
	if err != nil {
		return 0, nil, err
	}
	z.zitiNet.SetDeadline(time.Now().Add(5 * time.Second))
	log.Print("PACKEEEEEEEEEEEEET ", string(t), " ", string(s))
	// Addressing is abstract in Ziti, so return a dummy address
	// return n, &net.UDPAddr{IP: net.IPv4zero, Port: 0}, nil

	n, addr, err := z.net.ReadFrom(b)
	log.Print("OLDPACKEEEEEEEEEEEET ", string(b), " ", string(n))
	return n, addr, err
}

func (z *zitiPacketConn) WriteTo(b []byte, addr net.Addr) (int, error) {
	// Write data to the Ziti connection
	log.Print("WRIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIITE TO ", addr)

	fallback := &openziti.FallbackDialer{
		UnderlayDialer: &net.Dialer{},
	}
	dialer := openziti.ZitiContexts.NewDialerWithFallback(context.Background(), fallback)

	// Dial the Ziti service
	log.Print("ooooooooooooooooooooo ", addr.String())
	conn, err := dialer.Dial(z.network, addr.String())
	if err != nil {
		return 0, err
	}
	i, err := conn.Write(b)
	if err != nil {
		log.Print(err)
	}
	i, err = z.net.WriteTo(b, addr)
	return i, err
}

func (z *zitiPacketConn) Close() error {
	// Close the Ziti session
	return z.zitiNet.Close()
}

func (z *zitiPacketConn) LocalAddr() net.Addr {
	// Return a placeholder address; Ziti abstracts this
	return &net.UDPAddr{IP: net.IPv4zero, Port: 0}
}

func (z *zitiPacketConn) SetDeadline(t time.Time) error {
	return z.zitiNet.SetDeadline(t)
}

func (z *zitiPacketConn) SetReadDeadline(t time.Time) error {
	return z.zitiNet.SetReadDeadline(t)
}

func (z *zitiPacketConn) SetWriteDeadline(t time.Time) error {
	return z.zitiNet.SetWriteDeadline(t)
}

// ListenPacket announces on the local network address.
func (n *Net) ListenPacket(network string, address string) (net.PacketConn, error) {
	fallback := &openziti.FallbackDialer{
		UnderlayDialer: &net.Dialer{},
	}
	dialer := openziti.ZitiContexts.NewDialerWithFallback(context.Background(), fallback)

	// Dial the Ziti service
	log.Print("iiiiiiiiiiiiiiiii ", address, network)
	conn, err := dialer.Dial(network, "12.34.56.78:3478")
	if err != nil {
		return nil, err
	}
	log.Print("4444444444444444")

	ne, err := net.ListenPacket(network, address)
	if err != nil {
		return nil, err
	}

	return &zitiPacketConn{zitiNet: conn, net: ne, network: network}, nil

	// return net.ListenPacket(network, address)
}

If i turn on my local ziti tunnel and try to run the end-user app, ReadFrom OLDPACKEEEEEEEEEEEET gets data. If i turn off local ziti tunnel neither OLDPACKEEEEEEEEEEEET or the PACKEEEEEEEEEEEEET gets any data.

Questions:
in WriteTo(:

	i, err := conn.Write(b)
	if err != nil {
		log.Print(err)
	}
	i, err = z.net.WriteTo(b, addr)

Do nets WriteTo and zitis Write, behave the same way in this context?
Same question about zitis read and nets ReadFrom

Oh -- ok cool. So you have a server app somewhere and it's already deployed. That helps my mental picutre, thank you.

So the answer to this is probably both "yes" and "no". Let me explain...

"Yes", it does sound like the communication path will end up using UDP, but it sounds like those UDP packet will be at the far end -- that "server" side, after it egresses from the overlay network.

"No", insofar as OpenZiti allows the developer to "just write bytes". UDP/TCP/IP -- those are all abstracted away from the developer. Instead, the developer relies on OpenZiti getting the bytes being written from the client, to the destination.

So, that means when you write bytes into the ziti connection, you aren't explicitly writing UDP/TPC/IP, you're just writing to OpenZiti's connection. When the bytes arrive at the far side, the tunneler will know where/how to send the bytes through the host.v1 config. For example, in this case, the tunneling app would know: "these bytes are UDP bytes and need to be sent to host:1.2.3.4 on port 5678" so the tunneling app will send the bytes via UDP to the host:port.

Then, if the server responds, the tunneler needs to be able to receive the UDP bytes from the server, and relay them back over the OpenZiti connection to the client side, again, not worrying about UDP/TCP/IP because OpenZiti handles that. The tunneling app just "writes bytes" back to the client.

I hope that helps? Let me know if you want a diagram, but I hope those words are enough to make sense?

Ok, so if you see this happen, it probably means some "ack" type message is not being received at the client. This is where it can get somewhat tricky. My expectation is that there is some other process somewhere waiting on an ICE server which probably needs to be satisifed since you aren't going to have an ICE server in this situation. OpenZiti is effectively acting as your ICE server. (I'm by no means an expert at STUN/ICE/TURN fwiw, but I know 'enough' to help point you in the right direction I think). Somewhere in the client there's probably some kind of "does this connection require ICE" check because some OTHER port probe isnt' succeeding. You'll probably have to figure out why/where/how those are being blocked is my guess/expectation.

This makes sense to me because with the tunnel on, whatever that process is doing the probing is succeeding because that traffic made it onto the local network stack and then was sent over the overlay to the other side and the response was relay'ed and handled by the tunnelers.

Are you referring to net.Conn WriteTo ? I think you are and if that's the case yes, an OpenZiti connection should work in the exact same way as a net.Conn.

Does this help clarify anything?

1 Like

Hello, This clears things up a lot!

So, that means when you write bytes into the ziti connection, you aren't explicitly writing UDP/TPC/IP, you're just writing to OpenZiti's connection.

This works^

Then, if the server responds, the tunneler needs to be able to receive the UDP bytes from the server, and relay them back over the OpenZiti connection to the client side

This does not.

I made a simple ping pong example:

	// Dial the Ziti service
	conn, err := dialer.Dial("udp", "55.55.55.55:12345")
	if err != nil {
		log.Print("Anomalllyyy ", err)
		return
	}

	_, err = conn.Write([]byte("testsjdsjjdskjksd"))
	if err != nil {
		log.Print(err)
		return
	}

	for {
		var b []byte
		conn.SetReadDeadline(time.Now().Add(100 * time.Millisecond))
		n, err := conn.Read(b)
		if err != nil {
			log.Print(err)
		}

		log.Printf("%d - %s", n, string(b))
		time.Sleep(1 * time.Second)
	}

On ziti router i run nc -u -l -p 12345 -s 0.0.0.0 and on ziti side i created the necessary services, policies, etc.

When i run the ping pong example, the write works(in nc's logs i can see the received testsjdsjjdskjksd), but when i write something in nc, the read does not work. my go code doesnt receive udp data.

EDIT: same with tcp
EDIT: there was an error in my ping pong code var b []byte, should have been b := make([]byte, 10).

Ping pong works as expected

Hi @CarlosHleb,

Well I've learned a few things today. Most notably is that nc is an unreliable tool and depending on your OS and nc version it may, or may not.... "work"...

I did the same thing as you, testing our stuff for your usecase and I was shocked to see that I couldn't get it to work either... After an hour or so, and after pestering another engineer, I started asking ChatGPT if there were other "netcats" out there -- there are. Each of them apparently behave differently.

I run WSL on my Windows machine and in that WSL I have an Ubuntu 22.04 install which comes with nc version OpenBSD netcat (Debian patchlevel 1.226-1ubuntu2) (found using nc -h). This version of nc will NOT allow the server to respond with data. So I moved my testing to AWS, where I have an Ubuntu machine, it has OpenBSD netcat (Debian patchlevel 1.218-4ubuntu1)... Also doesn't allow the server to send data back to the client... Tried it with my MSYS2 windows port.... OpenBSD netcat (Debian patchlevel 1) -- doesn't work...

So then I tried it with a Debian12 VM I have and there nc works as you'd expect because Debian12 is using "netcat traditional" and reports version [v1.10-47]. MacOS nc also seems to 'work'.

So, is it possible you're using a nc that's of that OpenBSD variant that "doesn't work'???

On Ubuntu, I found both of these will work the way you would EXPECT it to... :confused:

sudo apt install netcat-traditional   # Install traditional version
sudo apt install ncat                 # Install Nmap's Ncat

using ncat:

using nc.traditional -- note the slightly different syntax:

nc.traditional -l -u -k -p 1234 127.0.0.1

So could that also be your problem?

Thats odd. nc works for me(i am on ubuntu).

You can confirm that without ziti, you can run two different terminal windows and see the client send to the server properly, and the server reply? can you tell me what version of nc you have?

I am going to just put a pr up with an example you can look at anyway shortly... that doesn't rely on netcat... :slight_smile:

You can confirm that without ziti, you can run two different terminal windows and see the client send to the server properly, and the server reply?

Yes. with and without ziti. There was an mistake in my ping pong code, i added EDIT to the post.

netcat-openbsd/noble,now 1.226-1ubuntu2 amd64 [installed,automatic]

Well that's good i guess... :slight_smile: I'll put a working example PR up in a bit and you can have a look.

1 Like

Here is a fully working example. It's also entirely self-contained (you can run it locally).

It's the udp-offload-example branch from the sdk-golang repo. PR is at:

Have a look and see if I've missed the mark somehow?

1 Like

forgot to mention - readme is render-able with this link: sdk-golang/example/udp-offload/README.md at 50fdeea7ecddf82bc295bd5b303d367b31141230 · openziti/sdk-golang · GitHub

PR helps a lot!

Another questions: What about UDP multiplexing?
pion/webrtc uses a single net.PacketConn to receive bytes from multiple addresses. I replaced pions writeTo and ReadFrom with zitified version. It fails to ReadFrom data thats meant for different port. The weird thing is that when i turn on ziti tunnel, everything works..

I would think that for multiplexing in general, you would be using multiple OpenZiti connections/dials. net.Packetconn is the message-oriented abstraction for networking as opposed net.Conn which is a stream-based abstraction. I don't quite understand why or how that would be related to multiplexing?

I think maybe you just need to Dial multiple times? If you have steps to reproduce what you're doing, or can push a project somewhere, it'd probably help me help you better.

You can also just implement the net.PacketConn interface, similar to what pion does themselves:

Here's a full example that doesn't use ziti but illustrates the idea.... It stats up 3 clients that all talk to the same server. Dunno if this is helpful or not but I think it might...

package main

import (
	"fmt"
	"net"
	"sync"
	"time"
)

// PacketConnShadow wraps a net.Conn to emulate net.PacketConn
var _ net.PacketConn = (*PacketConnShadow)(nil)

type PacketConnShadow struct {
	net.Conn
}

// ReadFrom reads data from the connection
func (f *PacketConnShadow) ReadFrom(p []byte) (n int, addr net.Addr, err error) {
	n, err = f.Conn.Read(p)
	addr = f.Conn.RemoteAddr()
	return
}

// WriteTo writes data to the connection
func (f *PacketConnShadow) WriteTo(p []byte, _ net.Addr) (int, error) {
	return f.Conn.Write(p)
}

func main() {
	serverAddr := "127.0.0.1:12345"

	// Start the server
	go startServer(serverAddr)

	time.Sleep(1 * time.Second) // Give the server time to start

	// Start three clients
	var wg sync.WaitGroup
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go func(clientID int) {
			defer wg.Done()
			startClient(serverAddr, clientID)
		}(i)
	}

	wg.Wait() // Wait for all clients to finish
}

func startServer(addr string) {
	listener, err := net.Listen("tcp", addr)
	if err != nil {
		panic(err)
	}
	defer listener.Close()
	fmt.Println("Server is listening...")

	for {
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println("Error accepting connection:", err)
			continue
		}
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()
	packetConn := &PacketConnShadow{Conn: conn}

	// Messages to send
	messages := []string{"aaaa", "bbbb", "ccccc", "dddd", "eeee"}

	for _, msg := range messages {
		_, err := packetConn.WriteTo([]byte(msg), nil)
		if err != nil {
			fmt.Println("Error writing:", err)
			return
		}
		time.Sleep(500 * time.Millisecond) // Simulate delay between messages
	}
}

func startClient(serverAddr string, clientID int) {
	conn, err := net.Dial("tcp", serverAddr)
	if err != nil {
		panic(err)
	}
	defer conn.Close()

	packetConn := &PacketConnShadow{Conn: conn}

	// Receive multiple "packets"
	buf := make([]byte, 1024)
	for i := 0; i < 5; i++ {
		n, svrAddr, err := packetConn.ReadFrom(buf)
		if err != nil {
			panic(err)
		}
		fmt.Printf("Client %d received: %s from %s\n", clientID, string(buf[:n]), svrAddr)
		fmt.Printf("Client %d: Message received\n", clientID)
	}
}

Hello,

I prepared a bare bones example of what i am trying to do: GitHub - CarlosHleb/ziti-livekit-example

The goal is to not have a ziti tunneler running on host, but use sdk instead.
The most of zitification is happening here in lib/pion-transport/stdnet/net.go

If tuneler is turned off on host, the publisher gets to Channel binding created(exchanges 4 messages over ziti to livekit), but then when it is supposed to connect it just timesout.

livekit logs says: added ICE candidate, after it timesout.

Thanks for the repo @CarlosHleb, that should really help me try to help you. I'll have a look and see if I can help out. It might take me a little while to consume it and try it etc but I'll get back to you soon...

1 Like

This would also be helpful for me, Thank you!

I have some time today so I thought I'd take a look... @CarlosHleb I got hung up when sudo was necessary to copy some certs out of the docker container. Is this strictly necessary? I'm going to skip that step and see where it brings me, but figured I'd ask here in case you can save me some time/heartache later :slight_smile:

# Add ziti-edge-controller-root-ca to host trusted crt's
dockercomp cp ziti-controller:/persistent/pki/ziti-edge-controller-root-ca/certs/ziti-edge-controller-root-ca.cert \
  ./store/ziti-edge-controller-root-ca.crt
sudo cp ./store/ziti-edge-controller-root-ca.crt /usr/local/share/ca-certificates
sudo cp ./store/livekit.crt /usr/local/share/ca-certificates
sudo cp ./store/turn.crt /usr/local/share/ca-certificates
sudo update-ca-certificates -f

EDIT:

I've gotten far enough to understand why you did this. I'm gonna try to work around it