{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z","url":"https://zyvop.com/kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z","title":"Kafka Connect and Kafka Streams: Moving and Processing Data","subtitle":"Move data in and out of Kafka without writing custom code, then transform and aggregate it in flight with Kafka Streams.","tldr":"Most Kafka projects need to pull data from databases and push it to other systems. Learn how Kafka Connect handles integrations, how Kafka Streams processes events as they arrive, and when to choose each one.","keywords":["Data Pipelines","Kafka Connect","ETL","Kafka Streams","Stream Processing","Tutorial","Apache Kafka from Zero to Production"],"entities":["Samod Alex","Data Pipelines","Kafka Connect","ETL","Kafka Streams","Stream Processing","Tutorial","Apache Kafka from Zero to Production","ZyVOP"],"keyTakeaways":["Part 6 of Apache Kafka from Zero to Production.","Most Kafka projects need three things beyond the basics: getting data in from existing systems, processing it as it arrives, and sending results out to other systems.","You could write producers and consumers by hand for all of it, but Kafka ships two tools built for exactly these jobs."],"headings":["The pipeline at a glance","Kafka Connect: integrations without custom code","Try it: start a Connect worker","Kafka Streams: processing as events arrive","Choosing the right tool","Settings at a glance","Mistakes to avoid","Next in the series","Frequently Asked Questions","What does Kafka Connect do in a data pipeline?","How is Kafka Streams different from Kafka Connect?","What are the key components of a Kafka Connect deployment?","How does Kafka Streams store state for stateful operations?","Which setting controls the maximum parallel tasks for a connector?","How can a sink connector handle bad records without stopping the job?","What does the processing.guarantee configuration in Kafka Streams affect?","What are common pitfalls to avoid when using Kafka Connect and Kafka Streams?"],"outboundLinks":[],"contentText":"Part 6 of Apache Kafka from Zero to Production. Most Kafka projects need three things beyond the basics: getting data in from existing systems, processing it as it arrives, and sending results out to other systems. You could write producers and consumers by hand for all of it, but Kafka ships two tools built for exactly these jobs. Kafka Connect moves data in and out, and Kafka Streams processes it in between. The pipeline at a glance A typical pipeline chains the two together: A source connector reads changes from a database and writes them to a Kafka topic. A Kafka Streams application reads that topic, transforms or aggregates the events, and writes the results to another topic. A sink connector reads the result topic and loads it into a warehouse, search index, or other system. The only code you write is the processing in step 2. Connect handles the rest through configuration. Kafka Connect: integrations without custom code Kafka Connect is a framework and runtime for streaming data between Kafka and other systems. Instead of building and operating your own producer or consumer for every database or storage service, you run a Connect cluster and give it a JSON configuration for each integration. Connect takes care of tracking progress, retrying after failures, and spreading work across machines. A few terms cover most of what you need: Worker: a Connect process. Workers that share a group.id form a cluster and divide the work, which is called distributed mode. A standalone mode also exists, but it is meant for development. Connector: the plugin that knows how to talk to one external system. A source connector reads from it into Kafka, and a sink connector writes from Kafka into it. Well-known examples include Debezium for capturing database changes, plus connectors for JDBC databases, object storage, and search engines. Task: the unit of parallelism. A connector splits its work into up to tasks.max tasks spread across the workers. Converter: controls how data is serialized in Kafka, for example JSON, Avro, Protobuf, or plain strings. Transform: a small single-message change applied in flight, such as renaming a field, masking a value, or routing to a different topic. Connect also stores its own state in Kafka. Source connector progress is kept in an offsets topic, and sink connectors track their position with ordinary consumer group offsets, so a restarted worker picks up where it left off. For bad records, a sink connector can be told to tolerate errors and send the failing records to a dead letter queue, a separate topic, instead of stopping the whole connector. That is controlled with errors.tolerance and errors.deadletterqueue.topic.name. Try it: start a Connect worker With your broker running, start a worker in distributed mode and ask its REST API what it can do: bin/connect-distributed.sh config/connect-distributed.properties # in another terminal curl localhost:8083/connector-plugins curl localhost:8083/connectorsThe first call lists the connector plugins the worker has loaded, and the second lists the connectors currently running. Plugins are installed by placing them on the worker's plugin.path. To start a connector, you send a POST request to /connectors with a JSON body holding a name and a config. The settings inside config that are specific to a connector come from that connector's own documentation. You can then check on it at /connectors/<name>/status. Kafka Streams: processing as events arrive Kafka Streams is a Java library for processing data in Kafka topics. It is not a separate cluster. You add it to your application, and the application becomes the stream processor. It reads from topics, transforms the data, and writes the results back to topics. It works with two ideas. A KStream is the full sequence of events, where every record matters. A KTable is the latest value for each key, like a table that updates as new records arrive. Counting orders per customer, for example, turns a stream of orders into a table of counts. Stateless operations such as filter and map look at one record at a time. Stateful operations such as count, aggregate, windowed aggregations, and joins between streams need memory of earlier events. Kafka Streams keeps that state in local stores and backs it up to internal changelog topics in Kafka. If an instance dies, another one rebuilds the state from the changelog. Setting num.standby.replicas above zero keeps warm copies ready for faster failover. Here is a complete application that counts orders per customer, using the orders topic with customer IDs as keys: Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, \"order-counter\"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, \"localhost:9092\"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); StreamsBuilder builder = new StreamsBuilder(); builder.<String, String>stream(\"orders\") .groupByKey() .count() .toStream() .to(\"orders-per-customer\", Produced.with(Serdes.String(), Serdes.Long())); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); Runtime.getRuntime().addShutdownHook(new Thread(streams::close));Scaling works the way consumer groups do in Part 3. Streams splits the input into tasks, one per input partition, and every instance you start with the same application.id takes a share of them. The input partition count is the limit on parallelism. And because Streams is built on the producer and consumer, the guarantees from Part 5 apply: setting processing.guarantee=exactly_once_v2 gives you exactly-once processing between Kafka topics with one line of configuration. Kafka Streams has limits worth knowing. It reads from and writes to Kafka only, so it relies on Connect for everything else, and it is a JVM library, so it fits Java and Scala applications. Choosing the right tool Tool Best for How it runs How you use it Kafka Connect Moving data between Kafka and external systems A cluster of workers JSON configuration, with no code for existing connectors Kafka Streams Transforming, aggregating, and joining data in Kafka topics A library inside your own application Java or Scala code Apache Flink Stream processing across many sources, often with SQL A separate cluster Java, Python, or SQL If you are copying data from one system to another, reach for Connect. If you are computing something from data already in Kafka, reach for Streams, or Flink if you want a separate processing cluster or SQL. Many pipelines use Connect and one processing tool together. Settings at a glance Setting Applies to What it controls Default tasks.max Connect Maximum parallel tasks for a connector 1 errors.tolerance Connect Whether a bad record stops the task or is skipped none (fail fast) application.id Streams Names the app, its consumer group, and its internal topics None (required) processing.guarantee Streams Delivery guarantee for processing at_least_once num.stream.threads Streams Processing threads per instance 1 num.standby.replicas Streams Warm copies of state for faster failover 0 Mistakes to avoid Writing a custom consumer and producer when a connector exists. You would rebuild offset tracking, retries, and scaling that Connect already provides. Running standalone mode in production. It has no failover, so use distributed mode. Mismatched converters. If a source writes Avro and the sink expects JSON, records fail. Keep converters consistent along the pipeline. Changing application.id casually. A new ID means a new consumer group and new state stores, so the application starts again from scratch. Joining topics that are not co-partitioned. Joins need both inputs to have the same number of partitions and the same partitioning by key. Running more instances than input partitions. The extras have no tasks to run. Next in the series This is Part 6 of Apache Kafka from Zero to Production. In Part 7, we cover running Kafka in production: sizing partitions, retention, security, monitoring, and managed options. Frequently Asked Questions What does Kafka Connect do in a data pipeline? Kafka Connect moves data between Kafka and external systems by using source and sink connectors, handling offset tracking, retries, and scaling via configuration without custom code. How is Kafka Streams different from Kafka Connect? Kafka Streams is a Java library that processes data inside Kafka topics; it runs inside your application, not as a separate cluster, and provides transformations, aggregations, and joins. What are the key components of a Kafka Connect deployment? A Connect deployment includes Workers (processes), Connectors (plugins for source or sink systems), Tasks (parallel work units), Converters (data serialization), and Transforms (single‑message modifications). How does Kafka Streams store state for stateful operations? It keeps state in local stores and backs them up to internal changelog topics in Kafka; optional warm replicas can be configured with num.standby.replicas for faster failover. Which setting controls the maximum parallel tasks for a connector? The tasks.max setting, which defaults to 1, defines the maximum number of parallel tasks a connector can create. How can a sink connector handle bad records without stopping the job? By setting errors.tolerance to skip or tolerate errors and specifying errors.deadletterqueue.topic.name, bad records are routed to a dead‑letter queue instead of halting the connector. What does the processing.guarantee configuration in Kafka Streams affect? It determines the delivery guarantee for processing, such as at_least_once or exactly_once_v2, ensuring the desired level of processing semantics between topics. What are common pitfalls to avoid when using Kafka Connect and Kafka Streams? Avoid writing custom producers/consumers when connectors exist, running Connect in standalone mode in production, mismatching converters, changing application.id casually, joining non‑co‑partitioned topics, and running more Stream instances than input partitions.","contentHash":"sha256:11507021c2ac6a01ec4933660c619393b133a09ac422f1d256b91b45b7fabfcc","authorName":"Samod Alex","authorUrl":"https://zyvop.com/author/samod","authorSameAs":[],"category":"Tutorial","tags":["Data Pipelines","Kafka Connect","ETL","Kafka Streams","Stream Processing"],"audience":"Software engineers and developers building applications with Tutorial","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":7,"wordCount":1573,"faqs":[{"question":"What does Kafka Connect do in a data pipeline?","answer":"Kafka Connect moves data between Kafka and external systems by using source and sink connectors, handling offset tracking, retries, and scaling via configuration without custom code."},{"question":"How is Kafka Streams different from Kafka Connect?","answer":"Kafka Streams is a Java library that processes data inside Kafka topics; it runs inside your application, not as a separate cluster, and provides transformations, aggregations, and joins."},{"question":"What are the key components of a Kafka Connect deployment?","answer":"A Connect deployment includes Workers (processes), Connectors (plugins for source or sink systems), Tasks (parallel work units), Converters (data serialization), and Transforms (single‑message modifications)."},{"question":"How does Kafka Streams store state for stateful operations?","answer":"It keeps state in local stores and backs them up to internal changelog topics in Kafka; optional warm replicas can be configured with num.standby.replicas for faster failover."},{"question":"Which setting controls the maximum parallel tasks for a connector?","answer":"The tasks.max setting, which defaults to 1, defines the maximum number of parallel tasks a connector can create."},{"question":"How can a sink connector handle bad records without stopping the job?","answer":"By setting errors.tolerance to skip or tolerate errors and specifying errors.deadletterqueue.topic.name, bad records are routed to a dead‑letter queue instead of halting the connector."},{"question":"What does the processing.guarantee configuration in Kafka Streams affect?","answer":"It determines the delivery guarantee for processing, such as at_least_once or exactly_once_v2, ensuring the desired level of processing semantics between topics."},{"question":"What are common pitfalls to avoid when using Kafka Connect and Kafka Streams?","answer":"Avoid writing custom producers/consumers when connectors exist, running Connect in standalone mode in production, mismatching converters, changing application.id casually, joining non‑co‑partitioned topics, and running more Stream instances than input partitions."}],"primaryTopic":"Tutorial","publishedAt":"2026-10-16T07:50:00.226Z","updatedAt":"2026-10-17T06:05:10.451Z","canonicalUrl":"https://zyvop.com/kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z"}