HTTP / REST (JSON)
The generic connector for any JSON API, with paging and rate limits.
Source GA
What the HTTP / REST (JSON) connector does
The generic connector for any JSON API. Describe the base URL, authentication, paging style and rate limit in YAML, and Synchronoid does the rest. Most public registries and partner APIs need no code at all.
The per-source request budget is shared by every pipeline that uses the source, so adding a pipeline never pushes you over the provider’s quota.
Capabilities
- Bearer, basic, API key and OAuth 2 client-credentials authentication
- Offset, page-number, cursor and link-header paging
- Per-source rate limits that respect
Retry-After - Automatic retries with exponential backoff, then pause and alert
- JSONPath selectors for record extraction
- Raw responses stored for replay
- Request and response provenance on every value
Set it up in minutes
Describe the connection in synchronoid.yaml or in Python. Both produce the same source, and credentials always come from secrets.
sources:
company_registry:
connector: http_json
base_url: https://registry.example.org/api
auth:
type: bearer
token: ${secret:REGISTRY_TOKEN}
paging:
style: cursor
param: after
next: $.meta.next
records: $.items[*]
rate_limit:
per_minute: 120
retry:
max_attempts: 5
backoff: exponentialimport synchronoid as sy companies = sy.Source( name="company_registry", connector="http_json", base_url="https://registry.example.org/api", rate_limit=sy.RateLimit(per_minute=120), ) @sy.asset(layer="staged", source=companies, retry=sy.Retry(max_attempts=5)) def company_profiles(ctx, raw): for item in raw.records(): yield {"org_id": item["id"], "name": item["name"].strip()}
Permissions and prerequisites
| Item | Requirement |
|---|---|
| Protocol | HTTPS with a valid certificate (HTTP is available for private networking only) |
| Response format | JSON or JSON Lines |
| Credentials | Stored as secrets; never written to logs |
| Egress | Allow the Synchronoid egress IPs for your region |
| Provider quota | Set rate_limit to at most the provider’s documented limit |
Good to know before you start
- XML and HTML responses are not parsed; use a custom connector.
- Paging styles that depend on the previous response body beyond one field need a custom connector.
- Request signing schemes such as AWS SigV4 are not built in.
Need something we don’t cover?
Tell us what you are missing. Feedback from beta users decides what we build next, and you can follow progress on the roadmap.
Request a feature →Keep reading
Connect HTTP / REST (JSON) to Synchronoid
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.