Provenance

Provenance answers the question "why does this value look like this?". Every field of every record carries its source, the time it was fetched and the run that produced it, so you can trace any value back to the payload it came from.

The provenance model

Provenance is recorded per value, not per table or per run. If two fields in one record came from different sources or different fetches, each field shows its own origin.

How provenance is collected

You do not write provenance code for the common case. When a connector stores a payload in raw, it assigns the payload a reference. When your function yields a record from a payload, Synchronoid links each output field to the input fields it was read from. Merges in the published layer keep the link of the winning value.

Adding your own notes

Use ctx.provenance inside an asset to add context that a person will want when they investigate a value later.

@sy.asset(layer="staged", source=companies)
def company_profiles(ctx, raw):
    for item in raw.records():
        address = item["address"]
        if not address.get("country"):
            address["country"] = "DE"
            ctx.provenance.note(
                field="address.country",
                reason="defaulted: provider omits country for domestic addresses",
            )
        yield {"org_id": item["id"], "address": address}

Querying provenance with the SDK

Use the client to read published records together with their provenance.

import synchronoid as sy

client = sy.Client()   # uses the token from `syn login` or SYN_TOKEN

record = client.assets.get("companies_current").record("org_1042", with_provenance=True)

prov = record.provenance["address.street"]
print(prov.source, prov.fetched_at, prov.run_id)

# Fetch the raw payload that the value came from
payload = client.raw.get(prov.payload_ref)
print(payload.json()["address"]["street"])

# Full history of one field
for version in record.history("address.street"):
    print(version.value, version.observed_since, version.run_id)

Querying provenance with the API

The REST API accepts include=provenance on record requests. See the REST API reference for all parameters.

$ curl -s "https://api.synchronoid.com/v1/assets/companies_current/records/org_1042?include=provenance" \
    -H "Authorization: Bearer $SYN_TOKEN"

The response contains the record, and a provenance object keyed by field path:

{
  "key": {"org_id": "org_1042"},
  "values": {
    "name": "Harbourview Holdings GmbH",
    "address": { "street": "Hafenstraße 12", "city": "Hamburg" }
  },
  "provenance": {
    "name": {
      "source": "company_registry",
      "fetched_at": "2026-10-28T02:00:41Z",
      "run_id": "run_8f2c1a",
      "step": "company_profiles",
      "payload_ref": "raw://company_payloads/2026-10-28T02:00:41Z/p17",
      "pointer": "/name",
      "observed_since": "2026-07-02T02:00:38Z"
    },
    "address.street": {
      "source": "company_registry",
      "fetched_at": "2026-10-28T02:00:41Z",
      "run_id": "run_8f2c1a",
      "step": "company_profiles",
      "payload_ref": "raw://company_payloads/2026-10-28T02:00:41Z/p17",
      "pointer": "/address/street",
      "observed_since": "2026-10-28T02:00:41Z"
    }
  }
}

Here the street was first seen in the latest fetch, so it was probably changed recently. The name has been stable since July.

Provenance in the control room

In the control room, open any record and click a field to see the same information, a link to the run, and a diff against the previous version. From a run, you can also open the lineage view to see which records it touched.

Limits and retention

• Provenance metadata is kept as long as the record version exists in the published layer.
• payload_ref only resolves while the raw payload is retained. After that the reference is still shown, but fetching it returns 410 Gone.
• Provenance adds a small amount of storage per field. You can turn it off for an asset with provenance=False, although we do not recommend it.

To read more about why we chose this model, see Provenance for every value.