Add a source with configuration, not a rewrite
A connector knows how to talk to one kind of system. A source is a configured use of it. Add the tenth source with ten lines of config, and keep the core of your platform untouched.
One connector, many sources
The http_json connector handles paging, authentication and error mapping once. Point it at three registries, five partner APIs or fifty customer accounts, and each source gets its own credentials, schedule and request budget.
- Declare sources in Python or in
synchronoid.yaml. - Secrets are referenced by name and never stored in code.
- Every source is visible, pausable and measurable in the control room.
sources: company_registry: connector: http_json base_url: https://registry.example.org/api auth: { type: bearer, token: ${secret:REGISTRY_TOKEN} } rate_limit: { per_minute: 120 } partner_listings_eu: connector: http_json base_url: https://partner.example.com/v2 auth: { type: bearer, token: ${secret:PARTNER_EU_KEY} } rate_limit: { per_minute: 60 }
Write a connector in an afternoon
When a system is not in the built-in list, implement two small methods: how to fetch one page and how to check the connection. Paging, retries, budgets, raw storage and provenance come from the SDK.
import synchronoid as sy from synchronoid.connectors import Connector, Page @sy.connector(name="registry") class RegistryConnector(Connector): config = {"base_url": str} def fetch(self, request, cursor=None) -> Page: resp = self.http.get(f"{self.config['base_url']}/companies", params={"after": cursor}) body = resp.json() return Page(records=body["items"], next_cursor=body.get("next")) def check(self) -> bool: return self.http.get(f"{self.config['base_url']}/health").ok
Respect every provider's limits
Public APIs have quotas. Several pipelines calling the same source must not add up to a ban.
Shared rate-limit budgets
All pipelines that use a source draw from one request budget, for example sy.RateLimit(per_minute=120). Excess work waits in a queue instead of failing.
Raw payload retention
Every response is stored untouched in the raw layer. When you fix a bug in your parsing, replay from raw without calling the provider again.
Backoff that listens
Connectors read Retry-After and quota headers and slow down before the provider has to tell you twice.
Start with what you already use
Databases, warehouses, files, streams and alert channels. Detail pages are available for the most used ones.
Ready to stop babysitting pipelines?
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.