Synchronoid documentation

Synchronoid is a managed platform for building, running and observing data pipelines. You describe sources and assets in Python, and the platform takes care of retries, rate limits, storage and provenance.

What is Synchronoid?

A Synchronoid project is a Python package with a synchronoid.yaml file. Inside it you define assets: pieces of data that you want to exist and stay current. You declare where the data comes from (a source), how it is transformed, and when it should be refreshed (a schedule or a sensor). Synchronoid runs the work and records what happened.

Four properties shape the whole product:

• Replay-safe steps. Every step is idempotent. Running it twice never duplicates data, and a run that loses its worker resumes from the last finished step.
• Three storage layers. Data moves from raw to staged to published, so you can always replay from the original payloads.
• Provenance on every value. Each field records its source, fetch time and run id.
• Sources by configuration. A new source is a short config and, if needed, a small connector. The core never changes.

Where to start

Synchronoid in 60 seconds

This is a complete pipeline. It declares a source, turns raw payloads into typed records, merges them into a current table with a change history, and runs every night.

import synchronoid as sy

companies = sy.Source(
    name="company_registry",
    connector="http_json",
    base_url="https://registry.example.org/api",
    rate_limit=sy.RateLimit(per_minute=120),
)

@sy.asset(layer="staged", source=companies, retry=sy.Retry(max_attempts=5))
def company_profiles(ctx, raw):
    for item in raw.records():
        yield {
            "org_id": item["id"],
            "name": item["name"].strip(),
            "address": item["address"],
        }

@sy.asset(layer="published", deps=[company_profiles])
def companies_current(ctx, company_profiles):
    return company_profiles.merge(key="org_id", track_changes=True)

nightly = sy.Schedule(cron="0 2 * * *", target=companies_current)

Run syn dev and open http://localhost:4040 to see the lineage graph and trigger a run. Run syn deploy to ship it to your workspace.

How the docs are organized

Getting access

All plans need an invite during the beta. If you do not have a workspace yet, you can request access. Questions about the docs can go through the contact form, or to the community.