Why Do API Rate Limits Keep Producing 429 Storms?

API rate limits are enforced with token buckets, not minute-long timers, so the 429 you keep hitting means your bucket is empty, not that the clock reset. The fix combines token-bucket awareness, Retry-After compliance, exponential backoff with jitter, and a hard retry budget.

What Does a 429 Actually Tell You?

A 429 is a signal that your request arrived when the server had no capacity left to serve it, and it is measured against a continuously refilled bucket, not a per-minute scoreboard. Most retry code assumes a counter that empties at the top of the minute. That assumption is wrong on every major provider, and it is why a well-behaved script still gets throttled.

The situation gets worse when multiple agents or services retry in lockstep. Amazon has put a number on this shape: five layers of services with three retries each can send 243 times the intended load at the bottom tier. The traffic that pushes you over the limit is often your own reaction to it, which is why api rate limits turn a speed bump into a wall.

Token Buckets: Why 60 Requests per Minute Is Not 60 at Once

A token bucket is the mechanism behind most api rate limits, and it takes two numbers to describe: the refill rate, which is your sustained throughput, and the bucket size, which is how much burst you can save up. Picture a machine that drips one token per second into a bowl holding 60. Arrive with a full bowl and you can spend everything at once; after that you get exactly what drips.

Anthropic states this directly: the Claude uses a token bucket algorithm, capacity is replenished continuously rather than reset at fixed intervals, and a rate of 60 requests per minute might be enforced as one request per second. A script that fires all 60 in one burst is over the limit, then idle, and it trips anyway.

You are rarely fighting a single bucket, either. Providers track several limits at once, and the status code alone does not tell you which one you hit.

Provider	What is counted	Scope
Anthropic	Requests per minute, input tokens per minute, output tokens per minute	Per organization tier
OpenAI	Requests per minute and per day, tokens per minute and per day, images per minute	Per organization and model
Gemini	Three quota dimensions applied per project, not per key	Per Google Cloud project
GitHub	Points (read = 1, write = 5), 100 concurrent requests, 90 seconds of CPU per 60 real seconds	Per repository or installation

Note the Gemini consequence: a second API key someone on your team created buys you nothing, because the quota follows the project. GitHub's secondary limit does not count requests at all; it counts points and CPU time.

Retry-After and the Header Free-For-All

The answer to a 429 is usually sitting in the response your error handler threw away. Providers attach headers that state the limit, the remaining quota, and the reset time, and the most important one is Retry-After, standardized in RFC 6585 in April 2012.

Anthropic version is the one worth taping to your monitor: the header gives the exact number of seconds to wait before retrying, and earlier retries will fail, not may fail. That number is the time your bucket needs to hold one more token. Treat it as an appointment, not a suggestion.

The complication is that the standard leaves room. RFC 9110 (2022) lets Retry-After be either a count of seconds or a full HTTP date, and both forms ship in production. Beyond that one header, encodings diverge: Anthropic sends reset times as RFC 3339 timestamps, OpenAI sends Go duration strings, GitHub sends epoch seconds, and Amazon uses its own millisecond field. Your parser has to handle several encodings of "come back later", and when a provider sends nothing, you fall back on a default that is older than the web itself.

Why Does Exponential Backoff Come from 1970?

Exponential backoff was invented for radio packets, not APIs. In 1970, Norman Abramson's ALOHAnet at the University of Hawaii wired campuses across the islands over a single shared radio frequency, where simultaneous transmissions destroyed each other and both sides retransmitted into another collision. The fix was to wait a random amount of time before retransmitting.

In 1976, Robert Metcalfe and David Boggs added the second half in their Ethernet work: each collision doubles the window you draw the random wait from. One collision means waiting up to two slots, two collisions up to four, and so on. Binary exponential backoff still runs in the silicon of every Ethernet interface.

The internet learned the same lesson at national scale in October 1986, when throughput between Lawrence Berkeley Laboratory and UC Berkeley fell from 32 kilobits per second to 40 bits per second, about 800 times slower over 400 yards. Van Jacobson spent nearly two years on the cause, because the hardware was fine; the software behavior on top of it had collapsed. The 1988 paper he published with Michael Karels, Congestion Avoidance and Control, remains the basis of TCP behavior today, and it argues that in a network of unknown topology with a constantly changing population of competing conversations, only exponential backoff has any hope of working.

The deep problem never changed. A retry is a bet that the failure was temporary. A 429 is the server telling you that it was not.

Jitter: Why Backoff Alone Still Arrives in Lockstep

Backoff fixes how long each client waits; it does nothing about when a whole fleet waits together. A thousand clients throttled in the same millisecond all back off 50 milliseconds and all come back in the same millisecond. You did not spread the load, you moved the spike and then doubled it in lockstep.

The fix is one line: do not sleep the full backoff, sleep a random amount between zero and the backoff. Mark Brooker published the arithmetic on the AWS Architecture Blog in March 2015. With a hundred contending clients, full jitter cut the calls needed to clear the entire set by more than half, and the formula later became the default in the AWS SDKs: delay equals a random value from zero up to the smaller of 20 seconds and the base delay doubled per retry.

The base delay should match what actually broke. AWS guidance distinguishes a connection reset, which waits around 50 milliseconds, from a throttle, which waits around 1,000 milliseconds, because a reset says something glitched while a 429 says the answer stays no until capacity returns.

How Do Retry Budgets and Adaptive Throttling Prevent 429 Storms?

Retry budgets cap the total work retries can add, because backoff and jitter only reshape the curve and never reduce how much you ask for. Google's SRE book prescribes a cap of three attempts per request and forbids letting retries exceed 10% of a client's own traffic; beyond that you are a load generator. The same book describes client-side adaptive throttling: each client tracks requests attempted versus backend-accepted over the last two minutes, and once requests are twice accepts, it rejects its own calls before they leave the process.

Amazon does the same job with an inward-pointing token bucket in the AWS SDK adaptive retry mode. The bucket limits retries, not requests: it starts at 500 tokens, a retry after a server error costs 14, a retry after a throttle costs 5, and a first-try success puts one back. At zero the client stops retrying on its own, and past roughly 22% sustained failures the budget drains faster than success refills it. Amazon's stated reason applies to everyone: continued retrying can delay recovery for everyone.

The failure mode these budgets prevent has a name. In 2021, four engineers led by Nathan Bronson published Metastable Failures in Distributed Systems: a system that will not recover even after the trigger is gone. Their arithmetic is simple. A database answering in under 100 milliseconds handles 280 queries per second comfortably out of a 300 capacity. A 10-second switch blip queues everything at once, and 280 requests become 560 with retries attached. The database slows, slowness causes timeouts, timeouts cause retries, and 560 stays 560 after the switch is fixed. Recovery does not begin at 300 served; it begins under 150. The paper calls that hidden capacity, the load you can climb back from as opposed to the load you can serve.

Both cloud operators have paid for this distinction publicly. The AWS US East 1 incident in October 2025, where a DNS race condition emptied a DynamoDB endpoint, did not end when the DNS record was fixed; a subsystem holding leases on every physical server had queued more work than it could finish, and AWS's own report named the state congestive collapse. Four months earlier, Google's global cloud outage write-up noted that a crashing binary looped across regions in part because service control lacked randomized exponential backoff, and one region took 2 hours and 40 minutes to recover not from the bug but from its own restart herd.

What Should You Actually Do, In Order?

Backoff and jitter are table stakes, not a strategy. The working order of operations, drawn from AWS and Google guidance, is:

1. Cap how many requests you allow in flight at once, so a fleet cannot converge into one spike.

1. Obey Retry-After exactly, and parse both the seconds and HTTP-date forms.

1. Apply exponential backoff with full jitter as your fallback when no header arrives.

1. Budget your retries with a client-side token bucket or adaptive throttle, and stop retrying when the budget empties.

1. Retry only idempotent operations. Amazon's own guidance is blunt: APIs with side effects are not safe to retry unless they provide idempotency. A read costs nothing twice; a charge costs your customer twice.

For transient faults like resets and server errors, retry fast and let jitter do its job. For a 429, the server has told you the failure was not temporary, so treat every retry as a capacity decision rather than a courtesy. The one question no dashboard answers is your hidden capacity: the load you can climb back from, not the limit you can serve.

Frequently Asked Questions

• Why do I get 429s even when I stay under my requests-per-minute quota? Because the limit is a token bucket with two parameters, not a per-minute counter. Your capacity refills continuously, so a burst that spends your saved-up tokens faster than the refill rate gets throttled even when your average looks compliant. Anthropic documents this behavior explicitly for the Claude.

• What is full jitter and why does it matter? Full jitter means sleeping a random duration between zero and your exponential backoff instead of the full backoff. Mark Brooker's 2015 analysis showed it cut the calls needed to clear a hundred contending clients by more than half, and it is now the default in AWS SDK retry logic.

• Should I always follow the Retry-After header? Yes. The header tells you exactly when your bucket will hold one more token, and retrying earlier is documented to fail, not merely risk failure. Handle both legal formats from RFC 9110: a number of seconds and a full HTTP date.

• What is a metastable failure? It is an outage that persists after its original cause is gone, sustained by the system's own retry and timeout behavior. Bronson and colleagues named it in 2021, and AWS used the related term congestive collapse in its October 2025 US East 1 incident report.

• How many retries should I allow? Google's SRE book caps retries at three attempts per request and no more than 10% of a client's traffic. AWS's adaptive retry mode starts with a 500-token budget where a throttled retry costs 5 and a server-error retry costs 14, so sustained heavy failure stops the client from retrying on its own.

Turn Your Own Explainer Into an Article

The deepest idea in this piece is that a single log line, the 429, hides a fifty-year story about token buckets, radio packets and collapse. That kind of knowledge usually lives in conference talks and screen recordings, not in searchable text. The team behind Crazystack typescript content and the Dev doido channel builds tools for exactly that gap, and crazystack.com.br collects more of their engineering material.

If you have an explanation like this sitting in a YouTube video, Skala Blog can turn it into a structured, citable article: paste the URL, get the transcription, review the draft, and publish. That is the whole flow at Skala Blog.

Source video