I've been hosting my Foundry game using a share.zrok.io link created by the above setup. However, there are times where:
Some players get a "403 Forbidden" error when following the game link, while others are able to join fine;
Joining fine works, then the player's game starts not loading assets, and when the player tries to refresh they get the 403.
I don't know if these are helpful debugging steps, but I've tried regenerating my zrok token, regenerating the start-server.ps1 command through Powershell to create a new share.zrok.io link, and enabled/disabled zrok. These have had inconsistent success in fixing the 403.
Not the most tech savvy so let me know if there's other info needed for debugging. Thank y'all!
Hey @shoebyron and thanks for popping in here to share how it's going. I understand you're running a zrok public share and sharing the public web URL for the share with other players, and they sometimes get a "forbidden" error when they connect to your game server in a normal web browser. This is disruptive because, while they can join, some things don't work. It's not all players, just certain players that get this error.
Here's a couple of things it might be. I hope it helps narrow possible causes.
Your account may be hitting its usage limit. You can look in the myzrok account dashboard to see your usage and status. You can see your current shares and accesses and their metrics in the zrok console, which includes a record of usage.
FoundryVTT may be malfunctioning. The game server may be responding with a "forbidden" error for reasons internal to the game, or due to a malfunction. In that case, zrok faithfully transmits the error to the player's web browser.
Does that make you think of any patterns that could explain it or that it must be something else?
Do you know what the difference is between api.zrok.io and myzrok.io account wise? I can login to the former but not the latter.
I'm hosting locally, so I shouldn't be hitting account caps, right? (zrok self hosted says no data transfer limits.) Additionally, I don't believe it's a Foundry problem as other players are able to join without 403, and it runs fine locally.
I doubt myzrok.io is relevant unless you've signed up for it. If you're not aware of what it is, don't worry about it.
It's definitely strange that some people are getting through fine while others are getting 403's. I think we'll have to have a deeper look with the team tomorrow. Is there any more details you can share? Do you know where your friends are located? How many people are joining? Is it always after say "5 friends" join, but then the last 3 fail?
Are there any other kinds of details you can share? Knowing the IP they are coming from could probably help us look on our side.
You shouldn' thave to regenerate your token -- this 403 sounds and feels to me like it's maybe related to the web application firewalls we have in place.
I'll bump this thread with our zrok team tomorrow and see what we can see. Thanks for reporting an issue here. It helps to have a better dialog than over on YouTube
Location: the issue has popped up when I'm trying to test out the join link (joining on a separate laptop connected to my own network), when we're all in the same room together (at a board game shop), and also when we're virtual in different houses across the state.
I'll note here that for the board game shop, that might be a red herring because we were using their WiFi and some network process of theirs might've gotten in the way. I bring it up because we got the server started successfully, as well as some joins, but the 403s happened intermittently like in the other cases.
Player count: we're 8 people. Because of party size, I initially assumed it was related to bandwidth/download rates, but now it happens sometimes when I'm the only person logged in during testing. There also wasn't consistency when players were getting kicked. Some players would successfully join, load a few assets, see images start failing to load, then get the 403 on a refresh.
What other details would help? I'm kind of iffy about IPs for privacy, especially my players, but can try to fill in other info.
Thank you for these details. All that information makes me think it's more likely something on the zrok.io site web protections in place. There's probably some kind of "web firewall" / limit getting hit that's not obvious. There are limits in place so if there are too many requests too quickly, which I am thinking is the issue, the it trips some kind of denial-of-service trigger on the web-hosting side. The thing that I think convinces me this is what's going on is when you said it even happens "when we're all in the same room together" (at the board game shop). That would mean all of you are accessing the same zrok share from the same IP address and downloading assets over HTTP servers (like Foundry VTT) makes a lot of requests.
I understand that. If you like, you could DM me here on this forum, that way you don't need to put it out on the internet for anyone to see :). You could also email me clint at openziti.org too if you want to send me one or more IPs for us to look at. Also, we might need your email address to look up your account and see if our logging catches anything.
Whatever it is, I appreciate you helping us track down the cause. I'll get with the people that know more about the web protections we have in place for zrok tomorrow ET and hopefully we can get back to you with some thoughts/ideas/questions to help clear this up.
What you're describing doesn't seem expected to me. We'll take a look and get back soon.
You indicated that you're self-hosting, but the errors that you're showing indicate that you are using the cloud service. If your service is accessible under *.share.zrok.io, then you are using the hosted cloud service, and it's subject to some level of rate limiting for request volume, which is possibly your issue if you're seeing this intermittently. I'd suggest trying a private share and seeing if that alleviates that problem.
We do protect the public shares endpoint with a rate limiter to prevent individual users from DDOS style attacks, so if there are more than a 5 requests per second coming in for several seconds from a single IP, that IP will become rate limited and only some requests will be able to get through.
Hi @TheLumberjack, I've gone ahead and emailed you the requested details. Thanks for the assist Let me know if the team finds anything interesting. Logs-wise, you'll want to look at the 7/4-7/7 and 7/12 dates if you can.
@mguthrie88 I'll take a look. Any recs to read up on the difference between hosting/share options? (Tbh, I don't know if this is my use case since I assume my players can't join over link.)
Knowing how FoundryVTT works, and knowing the video I made promotes using zrok public share, I doubt you'll want to flip over to private sharing. It is a bit more intricate than using the public share option as you'll have to setup TLS in Foundry, and your players will also need to run zrok. If you're interested in that, I can show you how, it does have the big benefit of bypassing any WAF/DDOS attack limits in place.
I was thinking about this some more, if you don't mind I'd be interested in knowing what would happen if one user joined, waited 15 seconds, another joined, waited 15 seconds, etc... I would think that'd 'slow roll' you past the "5 requests per second coming in for several seconds from a single IP" issue for the start, but after that, I'm not exactly sure what the communication pattern for foundry is like. When you move rooms or change scenes, I somewhat worry you'll possibly trigger the same limits.
Thanks again for the details, I got the email. I don't know how easy this is for you to trigger since it probably takes "5 or more friends"? But for logs, it would be interesting to see you run the FoundryVTT server in --headless mode and send the output to a file. To make it easier on you, I've pushed a start-server-with-logging.ps1 you can download and use just like you do the existing script.
When you run it (.\start-server-with-logging.ps1 -Public) you'll see a message telling you WHERE your log is (mine is at C:\Users\clint\AppData\Local\Temp\zrok-foundry-logs.txt) in blue and where your share is but it'll look a tiny bit different:
If you can reproduce the issue easily enough and then send along those logs (to email, ideally zipped) that'd be really helpful for us to help tune our protections.
Thanks again for helping out. We're glad you're enjoying zrok (mostly) We'll get it figured out I think....
Hey @shoebyron , now that I understand your use case better, I do think this is a really cool use case for zrok and we'll do what we can do help it work reliably. I was able to confirm that some of the FoundryVTT shares are sometimes tripping the rate limiter, but I would describe is as "just barely." We've made a few adjustments which will hopefully improve the experience and eliminate the 403s. Give it another try when you get a chance and let us know if this resolves your issue.
Y'all's support has been incredible and I'm super grateful for the tweaking your team has done behind the scenes to help make this work. My group is running a big session on Wednesday night, so I'll run the logging .ps1 during the session and send along the output. We'll try and trigger the 403 again. Regardless if the issues come up or if they're resolved, hopefully it'll provide some insight into the complications/successful resolution.
That sounds great. If you can, would you just post here when your session starts? Hopefully when you're done it will have been nothing but smooth sailing!
@TheLumberjack Clint, we're getting the 403 errors again. Admittedly, we all tried to join at the same time/really break it Some people are in, others are 403'd out.
Thanks for sending us the log. It's really helpful. Also glad to hear that waiting "a few minutes" allowed your other players to join in (that update was in the email).
Let me know how the rest of the session goes and if you hit any more 403's (i'd hope/expect you won't)
My players want to pass along their appreciation for what you and the zrok team are doing. Y'all have been amazing. If you learn anything else, let me know. This is a great project and I'd love to support it.
