Rate limits are a feature: sharing request budgets across pipelines

When you pull data from someone else's API, their rate limit is not an obstacle to route around. It is a contract. Pipelines that respect it are more reliable, and they make you a welcome customer. Here is how we think about request budgets.

The problem with per-pipeline limits

Most teams start with a simple approach: put a sleep between requests in each pipeline. It works while there is one pipeline per API. Then the team adds a second pipeline that uses the same key, and a nightly backfill, and a sensor that polls for changes. Each one is polite on its own. Together they send five times the allowed traffic, the provider answers with 429 Too Many Requests, and everything starts retrying at once.

The limit belongs to the source, not to the pipeline. So the budget has to be shared by everything that talks to that source.

Token buckets in plain words

A token bucket is a simple model. Imagine a bucket that holds a fixed number of tokens. Tokens are added at a steady rate, say two per second, up to the size of the bucket. Every request takes one token. If the bucket is empty, the request waits until a token appears.

Two numbers define it: the refill rate, which is your long-term average, and the bucket size, which is how large a burst you allow. A bucket of 10 with a refill of 2 per second lets you send 10 requests at once, and then 2 per second after that.

import time

class TokenBucket:
    def __init__(self, rate_per_sec, capacity):
        self.rate = rate_per_sec
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()

    def take(self, n=1):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens >= n:
            self.tokens -= n
            return 0.0                       # go now
        return (n - self.tokens) / self.rate  # seconds to wait

This version works inside one process. The hard part is making it work across many workers and many pipelines, because the bucket must live in one shared place.

One budget per source

In Synchronoid you declare the budget on the source. Every connector that uses the source draws from the same shared bucket, wherever its worker happens to run.

import synchronoid as sy

registry = sy.Source(
    name="company_registry",
    connector="http_json",
    base_url="https://registry.example.org/api",
    rate_limit=sy.RateLimit(per_minute=120, burst=10),
)

Now the scheduled sync, the backfill and the polling sensor all take tokens from the same 120-per-minute budget. If a backfill starts while the nightly run is active, they share. Nothing exceeds the quota, and nobody has to coordinate by hand. You can read the full options in the rate limits guide.

Backoff: what to do when the answer is still "no"

Even with a shared budget, you will sometimes get a 429 or a 503. The provider may have lowered your limit, or other clients share your key. The right response has three parts.

1. Obey the server. If the response has a Retry-After header, wait at least that long. It is the most accurate information you will get.
2. Back off exponentially. Without a header, wait 1 second, then 2, then 4, and so on, up to a ceiling.
3. Add jitter. Randomize each wait a little. If a hundred workers all retry at exactly the same moment, they cause the next failure themselves.

import random

def delay(attempt, base=1.0, cap=60.0, retry_after=None):
    if retry_after is not None:
        return retry_after
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)      # full jitter

In Synchronoid you do not write this yourself. You set the policy, and the platform applies it, including feeding Retry-After back into the shared bucket so that every pipeline slows down together:

@sy.asset(
    layer="staged",
    source=registry,
    retry=sy.Retry(max_attempts=5, backoff="exponential"),
)
def company_profiles(ctx, raw):
    ...

Know when to stop

Retrying forever is its own kind of abuse. If a source keeps failing, hammering it helps nobody. Synchronoid pauses a source that exhausts its retries repeatedly and alerts an operator, who can resume it with one command:

$ syn sources pause company_registry
$ syn sources resume company_registry

Being a good API citizen

Budgets and backoff are the mechanical part. A few habits go further.

• Read the provider's documentation and set your budget below the published limit, not at it. Leave room for other clients on the same key.
• Prefer incremental fetches. Ask for what changed since your last run instead of everything. It is faster for you and cheaper for them.
• Cache and deduplicate. Storing raw payloads means you can replay without calling the API again.
• Identify yourself. Send a clear user agent with a contact address so the provider can reach you instead of blocking you.
• Talk to the provider if you need more. A short email often gets a higher limit, and it costs less than a clever workaround.

Why this is a feature

A shared budget turns a fragile habit into a platform property. You get fewer incidents, because quota errors become waiting instead of failures. You get predictable runs, because backfills cannot starve live pipelines. And you get a relationship with your data providers that lasts. For public registries in particular, the teams who are polite are the teams who keep access.

If you want to see how budgets look in practice, the HTTP and REST connector page shows a complete example, and our post on idempotent pipelines explains why retrying is safe in the first place.