Bq
BigQuery
Write published tables and read query results from BigQuery.
Source Destination GA
Overview
What the BigQuery connector does
Publish tables to BigQuery with batch load jobs, or read query results as a source. Synchronoid creates and evolves tables for you and keeps every load replay-safe.
Partitioned and clustered tables are supported, so downstream queries scan less data and cost less.
Capabilities
- Batch load jobs with merge on your key
- Service account and workload identity authentication
- Time-partitioned and clustered tables
- Schema evolution for added columns
- Query results as sources, with incremental parameters
- Dataset location pinned to your chosen region
- Provenance exported as optional metadata columns
Configuration
Set it up in minutes
Describe the connection in synchronoid.yaml or in Python. Both produce the same source, and credentials always come from secrets.
synchronoid.yaml
destinations:
bq:
connector: bigquery
project: my-analytics-project
dataset: published
location: EU
credentials: ${secret:BQ_SERVICE_ACCOUNT}
publish:
- asset: meter_readings
table: readings
partition_by: reading_date
cluster_by: [meter_id]pipelines/bigquery.py
import synchronoid as sy bq = sy.Destination( connector="bigquery", project="my-analytics-project", dataset="published", location="EU", credentials=sy.Secret("BQ_SERVICE_ACCOUNT"), ) bq.publish( meter_readings, table="readings", partition_by="reading_date", cluster_by=["meter_id"], )
Requirements
Permissions and prerequisites
| Item | Requirement |
|---|---|
| Roles | BigQuery Data Editor on the dataset, BigQuery Job User on the project |
| Authentication | Service account key or workload identity federation |
| APIs | BigQuery API enabled in the project |
| Dataset location | Must match the location in the config |
| Storage staging | Optional bucket for large loads (needs Storage Object Admin) |
Tip. Create a dedicated, least-privilege account for Synchronoid. You can then revoke access in one step if you ever need to.
Limitations
Good to know before you start
- Streaming inserts are not used; loads are batched, so latency is minutes, not seconds.
- Changing a column type needs a manual migration.
- Tables with column-level security need the service account added to the policy tag.
Need something we don’t cover?
Tell us what you are missing. Feedback from beta users decides what we build next, and you can follow progress on the roadmap.
Request a feature →Related docs
Keep reading
Connect BigQuery to Synchronoid
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.