Recurring account-wide HTTP 429 across all active shares — share count ruled out

Recurring account-wide HTTP 429 across all active shares — share count ruled out, 5 other fixes tried, none resolved it

Setup:

  • 1 environment, normally 6 active public reserved shares (fixed subdomain names via zrok share public --subordinate -n public:<name>), each proxying a lightweight local HTTP service
  • Free tier, account verified with a card (no interstitial)
  • Usage per share: one small JSON poll roughly every 30 seconds — well under any published rate limit (2000 req/300s per IP, 7500 req/300s per share)
  • zrok v2.0.4 on Windows

Symptom: Every active share returns HTTP 429 simultaneously (never staggered — always all of them in the same short window, confirmed via each share's own access log showing zero gap or error right up until the moment the browser-side client starts getting 429 responses — the local tunnel process itself shows nothing wrong, meaning the rejection happens entirely at zrok's edge). This has recurred repeatedly across multiple days, sometimes several times in one day, gaps between incidents ranging from under a minute to several hours. Each incident usually clears on its own after 15-33 minutes provided nothing touches the account during that window (we added client-side logic to detect the pattern and go fully silent — no share create/delete calls at all — until it clears), but today one incident recurred less than 60 seconds after appearing to clear.

What we've tried, each a genuine attempt at a fix, none of which changed the pattern:

  1. Staggered our shares' startup (45s apart instead of near-simultaneous) — reduced simultaneous first-boot failures, but the recurring throttle still happens hours into steady operation, unrelated to startup timing
  2. Deleted 17 old orphaned/unreserved shares left over from earlier testing — no measurable change
  3. Removed a 7th, unrelated share entirely — no measurable change
  4. Verified the account with a card to rule out anti-abuse throttling on unverified accounts — no measurable change
  5. Found and killed a duplicate local recovery-automation process that had (unknown to us) been running two copies simultaneously, each independently capable of deleting/recreating shares on failure — a real bug, but the throttle recurred again ~18 minutes after confirming only one process remained and the account had zero duplicate/orphaned shares
  6. Reduced active shares from 6 to 3 and ran them in isolation for a controlled test, with no automation/recovery process running at all — all 3 shares still hit HTTP 429 simultaneously, at essentially the same elapsed time into the run as our 6-share setups typically do

What the zrok web console shows: normal-looking small activity graphs, no warnings, no near-limit indicators, account status "Free - Verified."

Question: With share count, orphaned resources, verification status, and our own automation all ruled out as the cause, is there a known rate limit or anti-abuse heuristic — not covered in the published per-IP/per-share request limits — that could produce a synchronized 429 across every active share on an account, independent of how many shares are active? Something tied to account age/history, a global per-account request ceiling regardless of share count, or a time-based pattern we're not seeing? Any pointers on what to check next, or logs/details that would help you diagnose it, would be appreciated.

Happy to share exact timestamps, share tokens, or full zrok2 overview output if useful.

Update: found the cause — it's per-IP, not per-account (following up on my earlier post about recurring account-wide 429s)

Posting an update on my earlier thread about a recurring HTTP 429 across all active shares, since I found the actual cause and it may help others hitting the same thing.

Short version: this was never an account-level issue. It's a per-IP throttle enforced upstream of zrok itself.

How I confirmed it:

Ran the exact same check — curl -i against one of my tunnels' /__broker_token endpoint — from two different networks within seconds of each other:

  • On my regular home connection: HTTP 429, empty body, Server: awselb/2.0
  • On a different network (mobile hotspot): HTTP 200 OK, normal JSON response, same account, same share, same everything else

The Server: awselb/2.0 header is the tell — that's an AWS Elastic Load Balancer response, not zrok's own application (zrok's real responses, including its own 429s, come back as JSON with an "ok" field). This rejection happens at AWS's edge, before the request ever reaches zrok's service.

Why my earlier tests were misleading: Everything I tried on the account side (verification, cleanup, reducing to 3 shares, fixing a local automation bug) correctly showed no effect, because none of it touched the actual variable: my public IP. My home ISP puts many customers behind a shared/rotating public IP (CGNAT-style), and apparently enough other traffic passes through some of those IPs to trip AWS's per-IP rate limiting on a regular basis — sometimes clearing in minutes, sometimes taking 20+ minutes, recurring multiple times a day, entirely independent of my own request volume (which I confirmed via browser dev tools is a tiny fraction of any published limit).

For anyone else hitting a similar pattern: If your 429s are erratic, don't correlate with your own request volume, and the response has no zrok-style JSON body — check the response headers for Server: awselb/2.0. If present, try the same request from a different network/IP. If it succeeds elsewhere, you're likely dealing with the same thing: an AWS-level rate limit tied to a shared IP, not zrok or your account at all.

Question for the zrok team, if anyone can confirm: is there a documented or typical threshold for this kind of per-IP limiting at the AWS layer? Even a rough number would help others self-diagnose faster than I did. Also curious whether zrok's own dashboard/console could ever surface "your requests are being rejected upstream of us" as a distinct signal from an account-level 429 — that distinction cost me a lot of debugging time.

Thanks to everyone who may have looked at the earlier thread — appreciate it.

Hi There,

Yes, in order to protect the platform so that one user doesn't tip over the whole platform for everyone else, we have two layers of rate limits in place.

IP-based - around 6.6 requests per second on a rolling window
Share based - around 25 requests per second (any IP).