Docs/Guides

Rate limits

Most data providers limit how many requests you can send. Synchronoid treats that limit as a budget that belongs to the source, not to a single pipeline. Every connector for the source draws from the same budget, so adding pipelines never pushes you over the quota.

How the shared budget works

When a step makes a request through a source, the connector first asks for a token. Tokens refill at the rate you configured. If none is available, the request waits in a queue. The budget is enforced across all workers and all runs in a workspace, including scheduled runs, sensors and backfills.

One budget per source, not per asset or per pipeline.
Waiting costs no step-run quota and does not count as a failed attempt.
If a source has several accounts, give each account its own source to get separate budgets.

Configure a limit

In Python, pass a sy.RateLimit to the source:

pipelines/sources.py

import synchronoid as sy

registry = sy.Source(
    name="company_registry",
    connector="http_json",
    base_url="https://registry.example.org/api",
    rate_limit=sy.RateLimit(
        per_minute=120,
        burst=20,
        reserve=0.2,
        max_wait="10m",
    ),
)

Or in synchronoid.yaml:

synchronoid.yaml

sources:
  company_registry:
    connector: http_json
    rate_limit:
      per_minute: 120
      burst: 20
      reserve: 0.2

Option	Meaning
`per_second`, `per_minute`, `per_hour`, `per_day`	Sustained rate. Use the unit that matches the provider documentation. You can combine several.
`burst`	Number of requests allowed at once after a quiet period. Defaults to one second of budget.
`reserve`	Fraction of the budget kept for scheduled runs. Backfills cannot use it.
`max_wait`	Longest time one request may wait. After that, the attempt fails with `SYN-2104` and is retried.
`concurrency`	Maximum requests in flight at the same time, for providers that limit parallel calls.

Set it below the real limitUse 80 to 90 percent of the quota the provider documents. Other tools or colleagues may use the same API key.

Observe quota usage

The Sources page in the control room shows requests per minute, queue length and average wait for each source. Click a source to see usage by asset and by run, which helps you find the pipeline that uses most of the budget. From the terminal:

terminal

$ syn sources status company_registry
Source           company_registry
Budget           120/min (reserve 20%)
Used (last 5m)   87/min avg, peak 118
Queued           14 requests, oldest waiting 38s
Provider 429s    0 in last hour

You can also read the same numbers from the REST API at GET /v1/sources/{name} and create an alert when the average wait stays high. See Alerts.

Handle 429 responses

Even with a correct budget, a provider may answer with 429 Too Many Requests. Synchronoid reacts in three steps:

It reads the Retry-After header, or the provider-specific header you configured, and stops all requests for the source until that time.
It lowers the effective rate by 25 percent and raises it slowly while no further 429 arrives.
It retries the failed request. The retry uses the http_429 rule of your retry policy.

pipelines/sources.py

rate_limit=sy.RateLimit(
    per_minute=120,
    retry_after_header="X-RateLimit-Reset",   # when the provider uses a custom header
    adaptive=True,
)

Repeated 429s pause the sourceIf 429 responses keep arriving after the rate was lowered, the source is paused and an operator is alerted. Check whether another system uses the same API key.

Using the limit in custom code

If a step calls a service without the built-in connector, ask for budget explicitly so the call is counted:

pipelines/enrich.py

@sy.step(source=registry)
def fetch_details(ctx, org_id):
    with ctx.source.acquire(cost=1):
        return requests.get(f"{BASE}/orgs/{org_id}", timeout=30).json()

Some endpoints cost more than others. Pass a larger cost for them, and the budget will be consumed accordingly.