Designing for API rate limits: budgets, back-off, and backfills that do not starve production

One budget, many consumers

Vendor limits apply per account, not per process. If four workers each implement their own limiter, the effective limit is exceeded four times over. Centralise the budget — a shared token bucket — and give each consumer a named allocation.

Back-off correctly

• Respect retry-after when the provider sends it; it is authoritative.
• Use exponential back-off with full jitter, otherwise every client retries in lockstep and re-creates the spike.
• Cap attempts and dead-letter the rest; infinite retry is a self-inflicted denial of service.
• Distinguish retryable errors from permanent ones. Retrying a 422 forever accomplishes nothing.

Backfills need their own lane

A one-off historical load will consume the entire budget if allowed. Run it at low priority with an explicit ceiling, checkpointed so it can be paused and resumed without restarting, and schedule it outside the peak window of the vendor's own limits.

Observe the budget

Emit consumed budget, 429 count, retry depth, and queue backlog age as metrics. Backlog age is the leading indicator: it rises before anything visibly breaks, which is exactly when intervention is cheap.