Synchronoid v0.14: change feeds, cost insights and new connectors

Version 0.14 is the biggest beta release since launch. Change feeds are now in beta for every workspace, cost insights arrive as a preview, there are four new connectors, and the syn command line tool is noticeably faster. Here is what changed. The full list is in the changelog.

Highlights

• Change feeds move from private testing to beta, with webhook, Kafka and pull delivery.
• Cost insights preview: see compute and request cost per pipeline and per source.
• New connectors: ClickHouse, DuckDB, Redshift and OpenTelemetry.
• CLI improvements: faster startup, syn runs replay --verify and better errors.

Change feeds in beta

Change detection has been part of Synchronoid for a while: we diff snapshots of a source and emit business-level events such as "company address changed". Until now, getting those events to your applications meant using the API directly. Change feeds fix that. A feed is a named, ordered stream of change events that downstream systems can subscribe to.

You can consume a feed in three ways: webhooks for simple integrations, Apache Kafka for high-volume streams, or a pull API when you want to control the pace. Every event carries the provenance of the new value, so a consumer can tell which source and run produced it.

feeds:
  company_address_changes:
    asset: companies_current
    events: ["company.address_changed"]
    kafka:
      brokers: ["kafka-1.internal:9093"]
      topic: "companies.address-changes"

Feeds are in beta for Team and Enterprise workspaces. Delivery is at-least-once with an ordering guarantee per record key, so consumers should be idempotent. We wrote about why in our post on replay-safe steps. See the change feeds guide and the product page for details.

Cost insights preview

The most common question from teams in the beta was "what does this pipeline cost?". Cost insights is our first answer. In the control room, each asset and source now shows step-runs, compute time and request counts, attributed to the pipeline that caused them.

The preview covers:

• Cost per pipeline and per source over the last 30 days.
• The most expensive steps, with a trend line.
• Request counts against each source's rate-limit budget.

It is a preview, so figures are estimates and may differ slightly from your invoice. We use it internally to find wasteful steps, and it found several of ours within a day. A longer description lives on the cost insights page. You can switch it on in synchronoid.yaml:

workspace: acme
insights:
  cost:
    enabled: true
    window_days: 30

New connectors

Browse all of them in the integrations directory. As always, a new source is a short configuration plus, where needed, a small connector.

CLI improvements

We spent a good part of this cycle on the command line, because it is where developers spend their time.

• Faster startup. syn now starts in about a third of the previous time, thanks to lazy loading of plugins.
• Verified replays. syn runs replay <run-id> --verify replays a run against a scratch copy and tells you whether anything changed.
• Better errors. Configuration mistakes now point to the file and line, and suggest a fix when the name is close to a valid one.
• Machine-readable output. syn runs list --json works everywhere, which makes scripting easier.

$ pip install --upgrade synchronoid
$ syn --version
syn 0.14.2
$ syn runs list --status failed --json | jq '.[].id'

Other changes

• The control room lineage graph can now be filtered by layer.
• Source pauses now record who paused them and why.
• Backfills show an estimated number of step-runs before you confirm.
• Several fixes to retry accounting for steps that were canceled mid-run.

Upgrading

Upgrading the SDK is a normal pip upgrade. Workspaces are updated by us, and there is nothing to migrate. One thing to check: the sy.Retry default for backoff is now "exponential" where it used to be "fixed". If you relied on the old default, set it explicitly.

The current patch release is v0.14.2. Tell us what you think through the community page, and see what is planned next on the roadmap.