Introducing Synchronoid

Today we are opening the private beta of Synchronoid. It is a managed platform for building, running and observing data pipelines, and it starts from one belief: a pipeline should be able to explain itself and survive a bad night without anyone waking up.

Where this started

Both of us have spent most of our careers building systems that pull data from places we do not control: partner APIs, public registries, exports from legacy systems, files dropped on an SFTP server at 3 a.m. The details differ, but the story is always the same. The first version is a script and a cron entry. It works. Then a second source arrives, then a tenth, and somebody asks why a customer's address in the dashboard is different from the one on their invoice.

At that point you discover what the script never knew: where a value came from, when it was fetched, whether the last run finished, and whether it is safe to run again. You end up writing that knowledge by hand, in every pipeline, slightly differently each time. We wrote it five times in five companies before we decided to write it once, properly.

What we heard from teams

Before writing any code we talked to about forty data teams. We asked them what they spend their time on. Almost nobody said "writing transformations". The answers clustered around four things:

• Babysitting. Re-running failed jobs, checking whether a retry created duplicates, pausing a source that is returning errors.
• Archaeology. Working out why a number looks wrong by reading old logs and diffing tables.
• Glue code. Every new source meant a new bespoke ingestion project, even when it was the tenth REST API of the same shape.
• Quota anxiety. Staying under a provider's request limits when five pipelines share one API key.

None of these are exciting problems, but together they decide whether a data platform is trusted or tolerated.

The ideas behind Synchronoid

Sources by configuration

Adding a source should be a short config and, at most, a small connector. The core of the platform does not change. One connector type can serve many sources and accounts, so the tenth REST API costs minutes rather than a sprint.

import synchronoid as sy

companies = sy.Source(
    name="company_registry",
    connector="http_json",
    base_url="https://registry.example.org/api",
    rate_limit=sy.RateLimit(per_minute=120),
)

@sy.asset(layer="staged", source=companies)
def company_profiles(ctx, raw):
    for item in raw.records():
        yield {"org_id": item["id"], "name": item["name"].strip()}

Provenance on every value

Every field in every record carries its source, fetch time and run id. When someone asks "why does this value look like this?", the answer is one query away instead of a day of digging. We wrote a whole post on how this works under the hood.

Three storage layers

Data lands in raw exactly as received, so you can always replay. It is then normalized into staged facts, and finally merged into published records with a change history. Each layer has one job, and you can rebuild any layer from the one before it.

Steps you can run twice

Every step is idempotent, and runs are durable. If a worker dies in the middle of a run, the run resumes from the last finished step. Retries use backoff and limits, and a source that keeps failing is paused automatically while an operator is alerted. Read more in our post on idempotent pipelines.

Where it is today

Synchronoid has been built with a small group of design partners over the last year, and today the first private beta release, v0.8.0, goes out to invited teams. The platform includes the orchestration engine, the connector framework, the three storage layers, durable runs and a first version of the control room. It does not include everything. We are still building change detection, change feeds, a lineage graph, cost reporting, deeper data-quality checks and more first-party connectors, and you can follow progress on the roadmap and in the changelog.

We want to be honest about the stage we are in. This is beta software. Interfaces will change, and we will tell you before they do. In return we ask for blunt feedback. The most useful thing a beta team can send us is a pipeline that Synchronoid handles badly.

Who it is for

If you maintain more than a handful of pipelines, depend on APIs with quotas, or need to answer questions about where data came from, this is built for you. Teams in fintech, property data, energy and logistics are already using it. You can read how Kestrel Property Data unified public registries and commercial feeds, or look at the product overview for the full picture.

How to join

All plans need an invite during the beta. The Explorer plan is free, and you can see the details on the pricing page. Request access and tell us a little about your data, and we will set up a workspace in the next batch, usually a few weeks later. If you would rather see it first, book a demo.

Thank you to everyone who argued with us about the design over the last year. We are looking forward to hearing what you build.

Lena and Tomás