Provenance for every value: how we store the why behind your data

Most data platforms can tell you which table a row lives in. Far fewer can tell you why a single field in that row has the value it has. This post explains how Synchronoid answers that question for every value, and what it costs us to do it.

The question we want to answer

Imagine a published record for a company. The address field says "Harbour Road 12". A user reports that the real address changed last month. To debug this you need to know: which source produced this value, when it was fetched, in which run, and whether another source disagreed. Table-level lineage tells you the published table was built from three upstream tables. That is useful, but it is not an answer.

We call the answer provenance: for each field, the source, the fetch time and the run id that produced the value currently stored.

The data model

A naive design adds three columns to every field: _source, _fetched_at, _run_id. For a record with 40 fields that is 120 extra columns, and almost all of them repeat the same few values. We went another way.

Provenance is stored as a small set of origin entries, and each field holds a short reference to one of them.

# One origin entry, shared by every field fetched in the same request
origin = {
    "id": "o_7f3a",
    "source": "company_registry",
    "fetched_at": "2026-06-02T02:03:11Z",
    "run_id": "run_b19c",
    "payload_ref": "raw/company_registry/9c1e04",
}

# Each field holds its value and a short reference to an origin
record = {
    "org_id": "org_4412",
    "name":    {"value": "Acme GmbH", "origin": "o_7f3a"},
    "address": {"value": "Hauptstrasse 12, Berlin",    "origin": "o_7f3a"},
}

All fields fetched in the same request share one origin, so a record usually carries one to three entries regardless of how many fields it has. The payload_ref points to the untouched payload in the raw layer, so you can go from a value to the exact bytes it came from.

Surviving merges

The interesting case is the published layer, where records from several runs and sources are merged. When a field changes, the new value takes its own origin, and the old value moves to the change history together with its origin. Nothing is overwritten without a trace.

Merging follows one rule: a field's provenance is always the provenance of the value that won. If two sources disagree, the merge policy picks a winner, and the loser is kept as a candidate with its own origin. You can see both in the control room.

import synchronoid as sy

@sy.asset(layer="published", deps=[company_profiles])
def companies_current(ctx, company_profiles):
    merged = company_profiles.merge(key="org_id", track_changes=True)
    ctx.log.info("merged companies with provenance")
    return merged

# Ask why a value looks the way it does
rec = sy.Client().assets.get("companies_current").record("org_4412", with_provenance=True)
prov = rec.provenance["address"]
print(prov.source, prov.fetched_at, prov.run_id)
# company_registry, fetched 2026-10-28 02:03 UTC, run run_b19c

What it costs

People are right to be suspicious of "metadata on every value". Here is what we measured on a mix of beta workloads, with records between 20 and 60 fields.

The reference per field costs a few bytes. The origin entries are small and shared, and they are stored once per request rather than once per field. Three further decisions keep the cost low:

• Interned origins. Identical origins are stored once and compared by id, so a million records from one run share a single entry.
• Columnar compression. The reference column in a batch has very few distinct values, which compresses extremely well.
• Raw payloads are not duplicated. Provenance points to the raw layer. It does not copy from it.

Querying provenance

Provenance is only useful if it is easy to ask. Besides the Python call above, the REST API returns it with any record when you ask for it:

$ curl -H "Authorization: Bearer syn_live_..." \
    "https://api.synchronoid.com/v1/assets/companies_current/records/org_4412?include=provenance"

The control room uses the same data to draw the lineage view: click a field, and you see the chain from the published value, through staged, back to the raw payload and the run that fetched it.

What provenance enables

• Debugging in minutes. A suspicious value leads straight to its run and payload.
• Auditability. Regulated teams can show where each reported number came from. Our fintech beta teams use this daily.
• Trust in change events. A change event says what changed and which source caused it.
• Safer source changes. When a provider alters a format, you can find every value that source produced and replay just those.

Trade-offs we accepted

Provenance is not free, and we made choices you should know about. Origins are immutable, so correcting one means writing a new origin. Computed fields, such as a normalized name derived from two inputs, record the origins of all their inputs, which makes them slightly larger. And provenance is retained as long as the value is retained, so very long history on very large datasets does cost real storage. You can set retention per layer.

We think this is the right trade. Storage is cheap compared with an afternoon spent explaining to a customer why a number is what it is. If you want to try it, the provenance documentation has a hands-on walkthrough.