Public & open data

Public registries and open APIs, kept in sync

Public data is free to use and hard to depend on. Formats change, rate limits are strict, and many sources only offer full snapshots. Synchronoid turns that into a steady stream of clean records and change events.

Request access Read the quickstart

The problem

Public sources change without warning

A registry renames a field, switches from CSV to JSON, or caps you at a few requests a minute. A nightly dump arrives late or truncated.

Compliance teams also need to show where each value came from and when it was fetched, and a folder of downloaded files does not do that.

Our approach

Budgets, snapshots and provenance

A shared request budget keeps you inside each provider's quota. Raw payloads are stored untouched, so a format change means updating a small connector and replaying, not re-downloading.

Snapshots are compared and turned into business-level events such as an address change. Every value keeps its source, fetch time and run id.

What you get

Made for sources you do not control

Rate-limit aware

Per-source budgets are shared across pipelines, so you never exceed a quota even when you backfill.

Snapshots to change events

Diff full snapshots and emit events your apps can subscribe to.

Provenance for compliance

Show the source, fetch time and run for any value in the published layer.

Survive format changes

Raw payloads are kept, so you fix the connector and replay without new requests.

Retries and pausing

Flaky endpoints are retried with backoff. A source that keeps failing is paused and an operator is alerted.

One model across sources

Normalize many registries and open APIs into one typed staged layer.

How it looks

From a nightly snapshot to change events

Declare the source and its budget, normalize the records, and let the published layer track changes for you.

Backfill history within the provider's limits
Pause a source with syn sources pause during an outage
Keep raw files for as long as your retention policy requires

pipelines/registry.py

import synchronoid as sy

registry = sy.Source(
    name="open_registry",
    connector="http_json",
    base_url="https://data.example.org/api",
    rate_limit=sy.RateLimit(per_minute=30),
)

@sy.asset(layer="staged", source=registry,
          retry=sy.Retry(max_attempts=5, backoff="exponential"))
def entities(ctx, raw):
    for r in raw.records():
        yield {"id": r["id"], "name": r["name"].strip()}

@sy.asset(layer="published", deps=[entities])
def entities_current(ctx, entities):
    return entities.merge(key="id", track_changes=True)

“Our upstream sources are slow, strict and change their formats a few times a year. With a shared request budget and stored raw payloads, those changes stopped being emergencies.”

IIngrid, Head of Data, Kestrel Property Data
Beta customer

Keep reading

Make public data dependable

Tell us which public sources you rely on. We will help you connect the first ones.

Request access Book a demo