Recurring account-wide HTTP 429 across all active shares — share count ruled out

zrok
1

Recurring account-wide HTTP 429 across all active shares — share count ruled out, 5 other fixes tried, none resolved it

Setup:

  • 1 environment, normally 6 active public reserved shares (fixed subdomain names via zrok share public --subordinate -n public:<name>), each proxying a lightweight local HTTP service
  • Free tier, account verified with a card (no interstitial)
  • Usage per share: one small JSON poll roughly every 30 seconds — well under any published rate limit (2000 req/300s per IP, 7500 req/300s per share)
  • zrok v2.0.4 on Windows

Symptom: Every active share returns HTTP 429 simultaneously (never staggered — always all of them in the same short window, confirmed via each share's own access log showing zero gap or error right up until the moment the browser-side client starts getting 429 responses — the local tunnel process itself shows nothing wrong, meaning the rejection happens entirely at zrok's edge). This has recurred repeatedly across multiple days, sometimes several times in one day, gaps between incidents ranging from under a minute to several hours. Each incident usually clears on its own after 15-33 minutes provided nothing touches the account during that window (we added client-side logic to detect the pattern and go fully silent — no share create/delete calls at all — until it clears), but today one incident recurred less than 60 seconds after appearing to clear.

What we've tried, each a genuine attempt at a fix, none of which changed the pattern:

  1. Staggered our shares' startup (45s apart instead of near-simultaneous) — reduced simultaneous first-boot failures, but the recurring throttle still happens hours into steady operation, unrelated to startup timing
  2. Deleted 17 old orphaned/unreserved shares left over from earlier testing — no measurable change
  3. Removed a 7th, unrelated share entirely — no measurable change
  4. Verified the account with a card to rule out anti-abuse throttling on unverified accounts — no measurable change
  5. Found and killed a duplicate local recovery-automation process that had (unknown to us) been running two copies simultaneously, each independently capable of deleting/recreating shares on failure — a real bug, but the throttle recurred again ~18 minutes after confirming only one process remained and the account had zero duplicate/orphaned shares
  6. Reduced active shares from 6 to 3 and ran them in isolation for a controlled test, with no automation/recovery process running at all — all 3 shares still hit HTTP 429 simultaneously, at essentially the same elapsed time into the run as our 6-share setups typically do

What the zrok web console shows: normal-looking small activity graphs, no warnings, no near-limit indicators, account status "Free - Verified."

Question: With share count, orphaned resources, verification status, and our own automation all ruled out as the cause, is there a known rate limit or anti-abuse heuristic — not covered in the published per-IP/per-share request limits — that could produce a synchronized 429 across every active share on an account, independent of how many shares are active? Something tied to account age/history, a global per-account request ceiling regardless of share count, or a time-based pattern we're not seeing? Any pointers on what to check next, or logs/details that would help you diagnose it, would be appreciated.

Happy to share exact timestamps, share tokens, or full zrok2 overview output if useful.

2

Update: found the cause — it's per-IP, not per-account (following up on my earlier post about recurring account-wide 429s)

Posting an update on my earlier thread about a recurring HTTP 429 across all active shares, since I found the actual cause and it may help others hitting the same thing.

Short version: this was never an account-level issue. It's a per-IP throttle enforced upstream of zrok itself.

How I confirmed it:

Ran the exact same check — curl -i against one of my tunnels' /__broker_token endpoint — from two different networks within seconds of each other:

  • On my regular home connection: HTTP 429, empty body, Server: awselb/2.0
  • On a different network (mobile hotspot): HTTP 200 OK, normal JSON response, same account, same share, same everything else

The Server: awselb/2.0 header is the tell — that's an AWS Elastic Load Balancer response, not zrok's own application (zrok's real responses, including its own 429s, come back as JSON with an "ok" field). This rejection happens at AWS's edge, before the request ever reaches zrok's service.

Why my earlier tests were misleading: Everything I tried on the account side (verification, cleanup, reducing to 3 shares, fixing a local automation bug) correctly showed no effect, because none of it touched the actual variable: my public IP. My home ISP puts many customers behind a shared/rotating public IP (CGNAT-style), and apparently enough other traffic passes through some of those IPs to trip AWS's per-IP rate limiting on a regular basis — sometimes clearing in minutes, sometimes taking 20+ minutes, recurring multiple times a day, entirely independent of my own request volume (which I confirmed via browser dev tools is a tiny fraction of any published limit).

For anyone else hitting a similar pattern: If your 429s are erratic, don't correlate with your own request volume, and the response has no zrok-style JSON body — check the response headers for Server: awselb/2.0. If present, try the same request from a different network/IP. If it succeeds elsewhere, you're likely dealing with the same thing: an AWS-level rate limit tied to a shared IP, not zrok or your account at all.

Question for the zrok team, if anyone can confirm: is there a documented or typical threshold for this kind of per-IP limiting at the AWS layer? Even a rough number would help others self-diagnose faster than I did. Also curious whether zrok's own dashboard/console could ever surface "your requests are being rejected upstream of us" as a distinct signal from an account-level 429 — that distinction cost me a lot of debugging time.

Thanks to everyone who may have looked at the earlier thread — appreciate it.

3

Hi There,

Yes, in order to protect the platform so that one user doesn't tip over the whole platform for everyone else, we have two layers of rate limits in place.

IP-based - around 6.6 requests per second on a rolling window
Share based - around 25 requests per second (any IP).

4

Thanks so much for the clear numbers — that confirms it exactly.

For anyone else who lands on this thread with the same symptoms: the ~6.6 req/s per-IP limit matches what I found independently by comparing responses across two different networks (see my earlier update in this thread — the Server: awselb/2.0 header was the giveaway that it's an edge-level rejection, not an account-level one).

My case is a home ISP that puts customers behind a shared/CGNAT public IP. My own traffic was nowhere near the limit, but other unrelated traffic sharing that same IP was apparently enough to tip it over at random times during the day — which lines up perfectly with a per-IP rolling window shared across everyone on that address, not anything about my account or usage.

Solution on my end is switching to a connection with a dedicated static IP, which should remove the "shared with strangers" variable entirely. No further ask from my side — just wanted to close the loop with the confirmed cause in case it saves someone else the same multi-day debugging path. Appreciate you taking the time to explain the limits clearly.

5

Separate issue today, different from the rate-limit thread above — posting here since it's the same account/setup.

Symptom: zrok2 share public --subordinate -n public:<name> ... fails consistently with:

{"error":"unable to create share: unable to create share: [POST /share][500] shareInternalServerError \"\"","msg":"error"}

Empty message body on the 500. Repeats identically on every retry (5+ attempts over ~30 min), across multiple different reserved share names on the same account, not just one.

What I've ruled out before landing here:

  • zrok status page (status.zrok.io): all green, no incidents logged today
  • DNS resolution to api-v2.zrok.io: working, resolves to 3 valid IPs
  • Client version: already on latest (v2.0.4, confirmed against GitHub releases)
  • Local zrok agent: restarted fresh, same error persists after restart
  • zrok2 overview: all reserved names present and intact (RESERVED: true), share tokens correctly show - (unbound) since nothing is currently running -- no corruption visible on my side
  • Direct API sanity check — curl -X POST https://api-v2.zrok.io/api/v2/share with no auth returns a clean, well-formed 401 with proper JSON body, so the API itself is reachable and responding normally to at least some requests

Earlier in the same session, before the agent restart, share creation was failing with a different, client-side error (json: cannot unmarshal object into Go value of type rest_model_zrok.ErrorMessage) — i.e. the server was sending back something my v2.0.4 client couldn't even parse as an error. After restarting the local agent, it changed to the clean, parseable 500 above. Mentioning both in case they're related on your end (e.g. two different error paths hitting the same underlying issue).

This looks like something only server-side logs would explain — happy to provide exact timestamps, the reserved share names involved, or anything else that would help you look it up. This account is also the one from the rate-limit thread above, for reference.

6

Just following up here to let you know in case you haven't seen.. Please follow that thread for updates

7

Hi. Just following up here too. I made this post on the pinned topic but just FYI here to make sure I cover all the bases... [duplicating the summary of that post here follows]

If you had stuck sessions that cannot be deleted, I'm told you should now be able to delete them on your own again. I don't have any I can test myself personally but if someone does and if they could post a follow-up here with success (or failure but let's hope not) that would be appreciated.

Thanks again for reporting the issues and bearing with us while we make zrok better!