Backfills

A backfill runs an asset for a range of past partitions. Use it to load history for a new source, to recompute data after a code change, or to fill a gap after an outage. This guide shows how to define partitions and how to run and tune a backfill.

Partitions

A partition is one slice of an asset, such as one day or one region. Partitions make runs small, make failures local and let Synchronoid track which slices are complete. You declare them on the asset.

import synchronoid as sy

daily = sy.Partitions.daily(start="2026-01-01")

@sy.asset(layer="staged", source=meters, partitions=daily)
def meter_readings(ctx, raw):
    day = ctx.partition          # "2026-03-14"
    for item in raw.records(date=day):
        yield {"meter_id": item["id"], "kwh": item["value"], "day": day}

Built-in partition types include sy.Partitions.daily(start=...), sy.Partitions.years(a, b) and sy.Partitions.static([...]) (a fixed list of keys, for example regions). Partition keys are always strings, and the same key always produces the same slice of data.

Run a backfill from the CLI

$ syn backfill meter_readings --from 2026-01-01 --to 2026-03-31
Backfill bf_41c7 planned: 90 partitions, concurrency 4

$ syn backfill meter_readings --partitions 2026-02-10,2026-02-11
$ syn backfill meter_readings --missing           # only partitions never completed
$ syn backfill meter_readings --failed            # only partitions that failed
$ syn backfill meter_readings --from 2026-01-01 --to 2026-01-31 --dry-run

Use --dry-run first. It prints the plan and an estimate of requests without starting anything. Add --downstream to also run the assets that depend on the selected one, in order.

Run a backfill from the control room

Open an asset, select the Partitions tab and choose a range or click individual cells. Press Backfill to see the same plan the CLI prints. Each cell shows the state of one partition: complete, running, failed or missing. You can cancel a backfill at any time. Running partitions finish, and queued ones are dropped.

Concurrency

A backfill runs several partitions at once. The default is 4. You can set it per backfill or as a default on the asset.

Backfills run at low priority, so a large load does not delay your scheduled runs. Both kinds of runs share the workspace limit on parallel steps.

Interaction with rate limits

All runs for a source, including backfills, draw from the same request budget. Raising concurrency does not make a backfill faster once the budget is used up. Extra partitions simply wait for quota.

The plan printed by --dry-run includes the earliest finish time at the current budget. A 90-day backfill that needs 54,000 requests at 120 requests per minute takes at least 7.5 hours, no matter how many workers you use.

Safety and idempotency

• A backfill is a normal set of runs. Each partition run is idempotent, so you can repeat or restart it. See Retries and replays.
• If a source gets paused automatically during a backfill, the backfill pauses too and continues after you resume the source.
• If raw data for a partition is still available, add --from-layer raw to avoid calling the provider again.

Next steps

• Learn how partitions relate to schedules in Schedules and sensors.
• See every flag in the CLI reference.