Storage layers

Every asset lives in one of three layers: raw, staged or published. Data flows in one direction, and each layer can be rebuilt from the one before it.

Overview

Raw

The raw layer is a faithful copy of what a source returned: the JSON body, the CSV file, the message batch. It also stores the request, the status code, the headers that matter and the fetch time. Nothing is parsed or cleaned.

Keeping raw data has two benefits. When your parsing code has a bug, you fix the code and replay from raw without calling the provider again. And when a provider changes or removes data, you still have what it said at the time.

@sy.asset(layer="raw", source=companies, retention=sy.Retention(days=180))
def company_payloads(ctx):
    return ctx.source.get("/companies", paginate=True)

Staged

A staged asset turns raw payloads into typed records. This is where you rename fields, fix types, trim strings and drop what you do not need. Each run writes a new set of records, tagged with the fetch it came from, so a run is a snapshot of the source at one point in time.

@sy.asset(layer="staged", deps=[company_payloads], schema=CompanyProfile)
def company_profiles(ctx, company_payloads):
    for item in company_payloads.records():
        yield CompanyProfile(
            org_id=item["id"],
            name=item["name"].strip(),
            address=item["address"],
        )

If a record does not match the schema, it is rejected and listed in the run, instead of silently corrupting later layers.

Published

The published layer is what downstream users read. It merges staged snapshots into one current record per key. With track_changes=True it also keeps every earlier version and who changed what, in which run.

@sy.asset(layer="published", deps=[company_profiles])
def companies_current(ctx, company_profiles):
    return company_profiles.merge(
        key="org_id",
        track_changes=True,
        on_missing="keep",   # do not delete records absent from a snapshot
    )

Change detection compares each new snapshot with the current record and emits events, such as "company address changed". These events feed change feeds. You can read the published layer through the REST API, with provenance attached to each value.

Replaying between layers

Because layers are derived from the one before, you can rebuild from any point.

# Rebuild staged and published from stored raw payloads, no source calls
$ syn materialize company_profiles --from-layer raw --downstream

# Replay one earlier run exactly
$ syn runs replay run_8f2c1a

Retention in practice

• Retention applies to data. Run metadata (status, timings, logs) is kept according to your plan's history, for example 7 days on Explorer and 90 days on Team.
• If raw data for a run has expired, you cannot replay it from raw. Replays use the oldest retained raw payload.
• Provenance records stay attached to published values even after the raw payload they refer to has expired.

Choosing a layer

If you are not sure where your asset belongs, ask what happens when it is wrong. A mistake in raw means the source was wrong. A mistake in staged means your parsing was wrong. A mistake in published means your merge rules were wrong. Each question has a different fix.