Every AI API has rate limits. Token-per-minute limits, requests-per-minute limits, daily caps, concurrent request limits — the specifics vary by provider but the constraint is universal. Building a pipeline that works in development and then fails unpredictably in production because of rate limiting is one of the most common deployment problems in AI engineering.
The right architecture treats rate limits as a normal operating condition rather than an exceptional one. This writeup covers the patterns that make pipelines resilient to rate limiting without requiring a large engineering investment.
Before implementing rate limit handling, it is worth understanding what the limit actually constrains. Token-per-minute limits are the most common for large language model APIs and are often the binding constraint for pipelines that generate long documents. A pipeline that generates a 10,000 token document will consume a large fraction of a standard tier's TPM budget in a single request.
Requests-per-minute limits are a separate constraint. A pipeline that makes many short calls — a planner call, several tool calls, a writer call, a verifier call — can hit RPM limits before hitting TPM limits even with modest token usage per call. Tracking both is necessary to diagnose which limit you are actually hitting.
Many providers also apply limits per API key rather than per account, which creates an opportunity: distributing load across multiple keys can effectively multiply your available capacity. This is the basis for the key rotation pattern.
Key rotation means maintaining a pool of API keys and cycling through them when one is rate-limited. The naive implementation is to rotate to the next key on error, but this breaks down when multiple concurrent pipeline runs are rotating independently — they all rotate to the same key at the same time, immediately rate-limiting it as well.
The correct implementation uses shared state to track the current key index. An atomic increment in Redis, modulo the number of available keys, distributes calls across the key pool in a round-robin pattern that is safe for concurrent access. When a specific key returns a 429, that key's index can be marked as temporarily unavailable and the rotation can skip it until the rate limit window expires.
Key rotation is not a substitute for proper rate limit handling — it is a multiplier on capacity. The retry and backoff logic still needs to exist for the case where all keys are exhausted or when operating with a single key.
Naive retry logic on rate limit errors makes the problem worse. If a service is returning 429s because it is at capacity, immediately retrying adds more requests to an already overloaded system. The Retry-After header, when present, tells you exactly how long to wait. When it is absent, exponential backoff with jitter is the standard approach.
Jitter — adding a random component to the wait time — prevents the thundering herd problem where multiple clients that were rate-limited at the same time all retry at exactly the same moment. A retry window of 2^n seconds plus a random fraction of that window gives enough spread to prevent synchronized retries while keeping wait times reasonable.
The retry budget should be finite. Retrying indefinitely on rate limit errors makes the pipeline's behavior unpredictable and can block the worker indefinitely. Two or three retries with increasing backoff, followed by a clean failure with a clear error message, is more predictable than infinite retries and easier to reason about from a user experience perspective.
Users who submit a task that hits rate limits deserve a better experience than a generic error message. The information that is actually useful to them is: the system is at capacity, here is approximately when it will be available again, and here is what will happen to their request in the meantime.
Tracking rate-limited providers in a shared store with a TTL matching the rate limit window gives you the ability to check capacity before accepting a task and to surface system status to users proactively rather than reactively. This lets users make informed decisions about whether to wait, switch providers, or come back later — rather than discovering the capacity problem partway through a five-minute pipeline run.
Supporting multiple AI providers — Gemini, Kimi, OpenAI, Anthropic — is often motivated by cost or capability differences, but it is also a reliability strategy. When one provider is at capacity, another may not be. A pipeline that can route to any supported provider based on current availability is more resilient than one locked to a single provider. For how this fits into the broader architecture of a production pipeline, see Building Multi-Agent Pipelines That Actually Work.
The tradeoff is consistency. Different providers produce different quality output on the same prompt. The system prompt that produces excellent output from one model may need adjustment for another. Provider-specific prompt variants add maintenance overhead. The right balance depends on how reliability-critical the application is and how much quality variance between providers is acceptable for the use case.