Kf
Apache Kafka
Consume topics as sources and publish change events to topics.
Source Destination Beta
Overview
What the Apache Kafka connector does
Consume Kafka topics as a source, or publish Synchronoid change events to topics for your downstream services. This is the same change feed you can get over webhooks or the pull API.
Consumers commit offsets only after a step finishes, so a worker failure never loses messages and a replay never duplicates them.
Capabilities
- Consumer groups with offset management by Synchronoid
- JSON, Avro and Protobuf messages with a schema registry
- Publish change events with the key you choose
- Exactly-once effect through idempotent steps
- SASL/SCRAM and mTLS authentication
- Back-pressure aware: slows down when downstream steps lag
- Consumer lag shown in the control room
Configuration
Set it up in minutes
Describe the connection in synchronoid.yaml or in Python. Both produce the same source, and credentials always come from secrets.
synchronoid.yaml
sources:
order_events:
connector: kafka
brokers: ["broker-1.example.com:9093"]
topic: orders.v1
group: synchronoid-orders
format: json
auth:
mechanism: SCRAM-SHA-512
username: synchronoid
password: ${secret:KAFKA_PASSWORD}pipelines/kafka.py
import synchronoid as sy order_events = sy.Source( name="order_events", connector="kafka", brokers=["broker-1.example.com:9093"], topic="orders.v1", group="synchronoid-orders", ) # To publish change events to Kafka, add a feed under feeds: in synchronoid.yaml.
Requirements
Permissions and prerequisites
| Item | Requirement |
|---|---|
| Kafka version | 2.8 or newer |
| ACLs (consume) | Read on the topic and consumer group, Describe on the topic |
| ACLs (publish) | Write and Describe on the target topic |
| Network | Public listener with TLS, or private networking (Enterprise) |
| Schema registry | Optional; Confluent-compatible API |
Tip. Create a dedicated, least-privilege account for Synchronoid. You can then revoke access in one step if you ever need to.
Limitations
Good to know before you start
- Beta: partition count changes while a run is active require a restart of the consumer.
- Messages over 5 MB are skipped and reported.
- Compacted topics are read from the current offset; full compaction replays are manual.
Need something we don’t cover?
Tell us what you are missing. Feedback from beta users decides what we build next, and you can follow progress on the roadmap.
Request a feature →Related docs
Keep reading
Connect Apache Kafka to Synchronoid
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.