Glossary

The language of data pipelines

Short definitions of the words you will meet in the docs, the control room and the API.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

A

Asset: A named piece of data that a pipeline produces, such as a table or a set of records. Assets declare their dependencies, which is how Synchronoid builds the lineage graph.

Backfill: A run that loads or recomputes data for past partitions, for example after adding a new source or fixing a bug. Backfills only touch the partitions you select.

Change data capture (CDC): A technique for reading only the rows that changed in a database, usually from its transaction log, instead of copying whole tables.
Change event: A business-level description of something that changed, such as "company address changed". Change events are created by comparing snapshots of a source.
Change feed: An ordered stream of change events that downstream applications subscribe to through webhooks, Kafka or the pull API.
Connector: A small piece of code that knows how to talk to one kind of system, such as HTTP/JSON or PostgreSQL. One connector can serve many sources.
Control room: The Synchronoid web app where you see runs, assets, lineage and alerts, and where you pause or resume sources.

Dead-letter queue: A place where records that could not be processed are stored so you can inspect and retry them later, without blocking the rest of the run.
Durable run: A run whose progress is saved after every step. If a worker stops, the run continues from the last finished step on another worker.

ELT: Extract, load, transform. Data is loaded into a warehouse first and transformed there, often with a tool such as dbt.
ETL: Extract, transform, load. Data is transformed before it is loaded into its destination.

Idempotency: The property that running an operation more than once has the same result as running it once. Every Synchronoid step is idempotent, so retries and replays never create duplicates.

Lineage: The map of how data flows from sources through assets to outputs. Lineage answers the question "what depends on this?".

Merge key: The field or fields that identify the same real-world entity across snapshots, such as org_id. Merging uses it to update existing records instead of adding new ones.

Partition: A slice of an asset, usually by date or by source account. Partitions can be run, retried and backfilled independently.
Pipeline: A group of steps and assets that run together, on a schedule or in response to a sensor.
Provenance: The recorded origin of a value: which source it came from, when it was fetched and which run produced it. Synchronoid stores provenance on every field.
Published layer: The layer that holds merged, current records together with their change history. Applications read from it through the API.

Rate limit: A cap on how many requests a provider allows in a period of time. Synchronoid connectors share a per-source request budget so quotas are never exceeded.
Raw layer: The layer that keeps untouched payloads exactly as received. It makes replay possible without calling the source again.
Replay: Running a finished run again from stored data. Because steps are idempotent, a replay fixes results without creating duplicates.
Retry policy: The rules for trying a failed step again: how many attempts, how long to wait and how the wait grows (backoff).
Run: One execution of a pipeline. A run has a unique id and a record of every step, log line and retry.

Schedule: A rule that starts a pipeline at set times, written as a cron expression such as 0 2 * * *.
Sensor: A small function that watches for an event, such as a new file in a bucket, and starts a pipeline when it happens.
Snapshot: The full state of a source at one point in time. Comparing two snapshots produces change events.
Source: A configured external system that data is fetched from, with its own credentials, schedule and request budget.
Staged layer: The layer that holds normalized, typed facts produced from raw payloads. Staged data is easy to validate and compare.
Step: The smallest unit of work in a run. Steps pass references to stored data, not large payloads.

Upsert: An operation that updates a record if it exists and inserts it if it does not. Upserts are a common way to make loads idempotent.

Webhook: An HTTP request that Synchronoid sends to your application when a change event or alert occurs.
Workspace: The top-level container for a team. It holds projects, users, sources, secrets and billing settings.

Missing a term? Tell us and we will add it. For concepts in depth, see the documentation.