PiKVM access with zrok

Support
1

Hey there

First off: Thank you very much for all the work that led to OpenZiti and zrok. It's amazing to have the ability to use such things. Even more amazing that it's transparent and open-source!

To the topic: I'm currently testing out remote administration using a PiKVM solution. For anyone that's interested and/or does not know what it is, have a look:

Now in short: This is a hardware and software based remote access solution. It's a Arch-Linux based Raspberry Pi that sends Keyboard Video Mouse over the network. Currently it is only exposed in my local network. I want it to stay that way.

That's where Open Ziti / zrok steps in. I wanted to use zrok to make the PiKVM available over the internet. The plan worked, I was able to successfully test it. Nonetheless, I have some open issues:

  1. Am I using the right backend mode and command?

I currently run "zrok share private --backend-mode tcpTunnel 192.168.8.126:443" on the Linux PiKVM. At the start, I thought that the default backend-mode proxy was the way to go. The PiKVM solution just runs a webserver on port 443 and is accessible using a web browser. However, the proxy backend did not work. I was only able to access the PiKVM using TcpTunnel mode.

  1. Performance problems
    As stated, the TcpTunnel works but it's not a fun experience. When controlling another computer using the PiKVM video stream, I have a delay of up to 20 seconds. So if I open a window, it takes around 10 to 20 seconds to show it. The mouse and keyboard actions are pretty instant, as expected. In total, I also feel that it's not as reliable as when using it directly on the same LAN: F.e. if I try to restart the video stream, it fails (when using zrok).

To help troubleshooting, I have noticed some unusual (I guess?) log entries on both the PiKVM and remote machine in the tunneler terminal. I've attached them here.

Screenshot 2024-12-26 112014
Screenshot 2024-12-26 1120141920Ă—1080 166 KB

If such a similar topic has already been discussed, please excuse the double-post. I'd appreciate if you can point me to that topic.

Otherwise, I'd be really happy to get some help.
I'm not that experienced in networking, so I might have missed some important things.

Thank you in advance & happy holidays.

BR
Peter

2

Hey Peter,

Do you get the same behavior when you restart both sides of the connection and establish fresh connections? Also, curious what your geo-location is (roughly). We have a global mesh, but if there is poor coverage in your region that might contribute to the latency.

3

Hey!

Thanks for the quick reply. Appreciate it.

Yes, I’ve tested this multiple times. Always the same. Just now, I performed another test and I was under the impression that it’s slightly better. But still not good to work with. WebRTC video mode also does still not work, so something might still be preventing proper video streaming.

I’m located in the north of Switzerland (currently).

BR

4

You're running zrok share private from a device in the same subnet as your PiKVM (probably 192.168.8.0/24). This allows you to tunnel a single TCP or UDP port. I suspect the problem is that, with WebRTC enabled, the web UI attempts to connect to the PiKVM's stream server on a separate UDP port.

If you can disable WebRTC in PiKVM settings, the web UI may function correctly over the single TCP port 443 you shared with zrok.

Another way is to share the PiKVM's IP address (all TCP and UDP ports) with a zrok VPN.

  1. Install the zrok-share Linux package on the PiKVM (based on the Linux install guide).

    # linux install example
curl -sSLf https://get.openziti.io/install.bash | sudo bash -s zrok-share

  2. Enable a zrok environment on the PiKVM, substituting your zrok account token for "mytoken."

    zrok enable "mytoken"

  3. Start the share on the PiKVM as root so zrok can manage the VPN network device. Closed permission mode ensures no other zrok accounts can access your share.

    sudo -E zrok share private --backend-mode vpn --closed

  4. Access the private share with elevated privileges on the client device. Run zrok with the administrator privilege on Windows or prefixed with sudo -E on macOS and Linux, if the zrok environment is not owned by root. Substitute the random private share token for "myshare."

    sudo -E zrok access private "myshare"

These steps are based on zrok VPN Guide | Zrok

If you like the way this works the next step is to follow the zrok "frontdoor" guide to automatically run the zrok share reserved command in headless/background mode after every reboot so it's always available. If you reach this point and need more specific guidance for your zrok-share.env file, just let me know!

5

Dear Ken

Thanks for taking the time and effort to help me.
I have read through your post carefully.

First up, one important thing: zrok already runs directly on the PiKVM. I installed the zrok binary there and ran zrok share private.

I now understand that WebRTC requires access to multiple different ports to work properly. A single TCP stream can not provide that.
Luckily, on the PiKVM, you can chose between H.264 / WebRTC and MJPEG / HTTP. So when I switch to MJPEG, the stream works and I can use the PiKVM. The issue, as already described, is the delay. Mouse movements and keyboard inputs work great and pretty instant, but the video stream from the HDMI output is really delayed.

I fear that the delay is introduced by the overlay network. Is there any way I can run a check or measure performance?

Furthermore I'm still confused on why the "proxy" backend mode does not work. AFAIK, the PiKVM is just running a webserver on port 443, so that should be fine right?

At last, I also thought about your proposed approach with the VPN backend. It's a good idea, but I wanted to keep the exposion of the host as limited as possible.
If I have understood the VPN backend mode correctly, I will make all ports available. I would rather only allow the ports needed and be as specific as possible. Just out of curiosity: Would it be possible to only share a specific port range? I know this from a commercial product (Netskope Private Access), where you can define "private apps", and only allow access to certain IP ranges and port ranges.

Looking forward to addtional insight or help if I'm wrong anywhere :slight_smile:

1 Like
6

I'm not quite sure what you meant about the instant mouse movements unless you had another way to see the mouse move while waiting for the delayed video frames. The main problem is the remote display is lagging too much, right?

I see PiKVM uses GitHub - pikvm/ustreamer: µStreamer - Lightweight and fast MJPEG-HTTP streamer to stream the remote display. I did a quick test by installing that package on my workstation, sourcing an HDMI device with v4l2, and sharing the built-in web server with zrok.io. Yep, very laggy video frames. There's no perceptible delay viewing the ustreamer feed through my local zrok dev instance. The delay is about 4s through my personal self-hosted zrok instance, which is ~85ms away. We may be seeing the same symptom.

There are many variables, but I know it's possible to achieve acceptable latency for live video with zrok because I've had natural conversations with Jitsi Meet and used graphical terminal services protocols like X11 SPICE, used the terminal in Cockpit's web UI, and SSH.

I'd start by tuning the display scale, compression, framerate, etc.,

to see if there's any button you can push that makes it worse or better. I considered running it through an HLS converter, e.g., Restreamer, but I'm unsure if achieving dynamic bitrate is possible without introducing a prohibitive delay.

I can't think of a better test than moving the cursor around on the remote display.

Yes. I used the proxy mode for my ustreamer test, but PiKVM might be using additional ports. Proxy mode will work as long as everything is HTTP on the same TCP port.

zrok's proxy and tunnel modes are limited to a single TCP or UDP port, and you correctly understood the VPN mode shares an IP address (all ports, protocols, etc.). Still, you can use a firewall to restrict network flows on the address.

OpenZiti, the overlay platform on which zrok is built, has tunnel configurations that can express port ranges, so zrok could gain that ability if implemented (link to roadmap).

Since you need more control over the data path, would you like to explore self hosting your own zrok instance? That way, you could place a router optimally to short circuit the path and minimize latency. I used this approach to run the controller in Docker on my VPS: Self-hosting guide for Docker | Zrok

You'd need to access the PiKVM through a zrok private share to optimize for latency because it's independent of the zrok public frontend and will use the nearest ziti router, usually.

7

Great update, amazing! :slight_smile:

Yes, I can tell that mouse and keyboard inputs are pretty instant since I watch it on the physical montior here. As you said, only the video streams are delayed.

I will try to play around with the settings, as you proposed.

I will also go for a self-hosted instance, as this would have been the final goal anyway. I only tested zrok, since it was easy to “setup”. In the end I wanted to go with self-hosting OpenZiti.

Would you recommend me to directly go with Open Ziti selfhosted? I thought that there were some more possiblities with OpenZitit.

8

Right, zrok simplifies OpenZiti services as "shares," and it sounds like you need more control, so I recommend self-hosting OpenZiti. This can be as minimal as one controller and one router to provide the control and data planes, then you'll have some OpenZiti identities added to your endpoint applications importing OpenZiti SDKs or tunnelers on those hosts as proxies for your applications.

Deployment guides: Deploying to Production | OpenZiti

For a test environment with Docker, this is the most simplified example. It includes the web console. ziti/quickstart/docker/all-in-one/README.md at main · openziti/ziti · GitHub

1 Like
9

Awesome! Thank you.
I will do a "POC", as soon as I have the time and report back.

As of now, you guys helped a lot and my questions are answered.
You can go and close this topic, if required (or tell me how to :slight_smile: )