S3
Amazon S3
Read files from buckets and keep raw payloads in your own bucket.
Source Destination GA
Overview
What the Amazon S3 connector does
Read CSV, JSON Lines, Parquet and compressed files from S3 buckets, and write exports back. Synchronoid can also keep its raw layer in a bucket you own, so payloads never leave your account.
New files are detected with a listing sensor or with S3 event notifications, and each file is processed exactly once per run.
Capabilities
- CSV, JSON Lines and Parquet readers
- Gzip and Zstandard decompression
- Prefix and pattern matching, such as
exports/*/data.csv.gz - Event-driven runs with S3 notifications through a sensor
- Bring-your-own-bucket for the raw layer
- Role assumption with an external id; no long-lived keys
- Server-side encryption with KMS keys
Configuration
Set it up in minutes
Describe the connection in synchronoid.yaml or in Python. Both produce the same source, and credentials always come from secrets.
synchronoid.yaml
sources:
partner_exports:
connector: s3
bucket: acme-partner-exports
prefix: daily/
pattern: "*.csv.gz"
format: csv
region: eu-central-1
role_arn: arn:aws:iam::123456789012:role/synchronoid-read
external_id: ${secret:S3_EXTERNAL_ID}pipelines/amazon_s3.py
import synchronoid as sy exports = sy.Source( name="partner_exports", connector="s3", bucket="acme-partner-exports", prefix="daily/", pattern="*.csv.gz", role_arn="arn:aws:iam::123456789012:role/synchronoid-read", ) @sy.asset(layer="staged", source=exports) def partner_orders(ctx, raw): for row in raw.records(): yield row @sy.sensor(target=partner_orders, every="5m") def new_export_file(ctx): latest = ctx.source.head("daily/latest.csv.gz") if latest.etag != ctx.cursor: ctx.cursor = latest.etag return sy.RunRequest(run_key=latest.etag)
Requirements
Permissions and prerequisites
| Item | Requirement |
|---|---|
| IAM permissions (read) | s3:ListBucket, s3:GetObject |
| IAM permissions (write) | s3:PutObject, s3:AbortMultipartUpload |
| Trust policy | Allow the Synchronoid account to assume the role with your external id |
| Bucket region | Any; same-region buckets avoid transfer costs |
| Encryption | SSE-S3 or SSE-KMS (add kms:Decrypt) |
Tip. Create a dedicated, least-privilege account for Synchronoid. You can then revoke access in one step if you ever need to.
Limitations
Good to know before you start
- Files are read whole; single files over 5 GB need partitioning at the source.
- Listing very large prefixes (millions of objects) is slow; use event notifications.
- Object versions are ignored; only the current version is read.
Need something we don’t cover?
Tell us what you are missing. Feedback from beta users decides what we build next, and you can follow progress on the roadmap.
Request a feature →Related docs
Keep reading
Connect Amazon S3 to Synchronoid
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.