Kafka, RabbitMQ, and Amazon SQS all move messages between the parts of a system, so they often get compared as if they were interchangeable. They are not. Each one is built around a different idea of what a message is and what should happen to it afterward, and that difference matters more than any benchmark. This guide compares them as of Kafka 4.2, RabbitMQ 4.x, and the current SQS documentation, and ends with a simple way to choose.
The short answer
Choose Kafka when many independent systems need to read the same stream of events, you want to replay history, or you need very high throughput and stream processing.
Choose RabbitMQ when you need flexible routing, per-message acknowledgements, priorities, or support for several messaging protocols, at moderate scale.
Choose Amazon SQS when you are on AWS and want a simple, fully managed queue for background work, with no servers to run.
Three different mental models
Kafka is a log. Producers append events to a durable, replicated log, and consumers read it at their own pace by tracking an offset. Reading a record does not remove it. It stays until the retention period ends, so any number of consumers can read the same data, and new ones can start from the beginning.
RabbitMQ is a smart router with queues. Producers publish to an exchange, which uses bindings and routing keys to decide which queues receive each message. Consumers take messages from queues and acknowledge them, and the broker then removes them. The broker does the thinking, and consumers stay simple.
Amazon SQS is a queue as a service. You create a queue, send messages to it over an API, and workers receive and delete them. There is no cluster to size or patch. AWS runs it, and you pay per request.
Side-by-side comparison
Kafka | RabbitMQ | Amazon SQS | |
|---|---|---|---|
Core model | Distributed, replicated log | Broker with exchanges and queues | Managed queue service |
What happens to messages | Kept for the retention period, whether or not they were read | Removed from a queue once acknowledged, but kept by streams | Deleted by the consumer after processing, or expire after at most 14 days |
Replay | Yes, by moving offsets | Only with streams | No |
Routing | Topics and keys, with consumers filtering | Exchanges, routing keys, and bindings | One queue per destination, with SNS for fan-out |
Ordering | Per partition | Per queue, weakened by competing consumers and redelivery | Best effort in Standard, strict per message group in FIFO |
Delivery | At least once, with exactly once inside Kafka | At least once with publisher confirms and acknowledgements | At least once in Standard, exactly-once processing in FIFO |
Scaling consumers | Up to the partition count per group, and beyond with share groups | Add competing consumers to a queue | Add consumers freely |
Operations | Heaviest, with managed services available | Moderate, with managed services available | None, fully managed on AWS |
Message lifetime and replay
This is the deepest difference. In Kafka, the log is the source of truth. A new analytics service can start reading from the first record, and a bug fix can be followed by reprocessing last week's events. That makes Kafka a natural fit for event sourcing, change data capture, and any system where history matters.
In a RabbitMQ queue and in SQS, a message exists to be handled once. When the consumer acknowledges it, or deletes it in SQS, it is gone. SQS keeps unprocessed messages for 4 days by default and up to 14 days at most. If a consumer fails to delete a message within the visibility timeout (30 seconds by default), the message becomes visible and is delivered again.
RabbitMQ also offers streams, an append-only log type where consumers read without removing anything and can start from any point. That gives it Kafka-style replay for the cases that need it.
Routing
RabbitMQ is the strongest here. Its four exchange types, direct, fanout, topic, and headers, let a producer publish once and have the broker deliver to the right queues, for example every message whose routing key matches order.*.eu. Request-reply and RPC-style patterns are well supported too.
Kafka routing is simpler. Producers choose a topic, the key decides the partition, and consumers subscribe to topics and filter in their own code or with Kafka Streams. SQS has no routing by itself. To fan out one message to several queues, you publish to an Amazon SNS topic that has SQS queues subscribed, optionally with filter policies.
Ordering and delivery guarantees
Kafka guarantees order within a partition, so a key such as a customer ID keeps one customer's events in order. Delivery is at least once by default, and transactions give exactly-once processing between Kafka topics.
RabbitMQ queues are first in, first out, but order can break when several consumers compete for messages or a message is redelivered. It offers at-least-once delivery through publisher confirms and consumer acknowledgements. It does not offer exactly-once.
SQS Standard queues deliver at least once and make a best effort at ordering. SQS FIFO queues keep strict order within a message group ID and deduplicate messages sent during a 5-minute window, which gives exactly-once processing. The trade-off is lower throughput limits than Standard queues.
Whichever you pick, write consumers so that processing a message twice is harmless. Duplicates can occur in all three.
Handling failures
Messages that keep failing need somewhere to go. SQS has this built in: a redrive policy moves a message to a dead letter queue after a set number of receives. RabbitMQ uses dead letter exchanges, and quorum queues in RabbitMQ 4.0 limit redelivery to 20 attempts by default. Plain Kafka consumers have no built-in dead letter queue, so applications publish failed records to a separate topic themselves, although Kafka Connect sinks can do it through configuration.
Scaling consumers
In RabbitMQ and SQS, adding more workers to a queue is easy. Each one takes the next available message. In Kafka, a consumer group can have at most one active consumer per partition, so the partition count caps how far one group can scale out.
Kafka 4.2 changed this for queue-style work. Its share groups, from the Queues for Kafka feature (KIP-932), are now production-ready. They let multiple consumers process records from the same partitions, with individual acknowledgement of each record and delivery counting, so consumers can outnumber partitions.
Operations and cost
Kafka is the most demanding to run. Even with KRaft, you manage brokers, disks, partitions, retention, and upgrades, or you pay a managed provider such as Confluent Cloud or Amazon MSK. RabbitMQ is lighter, and its quorum queues replicate data using Raft. Managed options include Amazon MQ and hosted services from several vendors.
SQS needs no operations at all. There are no brokers to size, and capacity scales for you. The costs of that convenience are lock-in to AWS and a feature set that is deliberately small: no replay, no routing, and no priorities. Messages can be up to 1 MiB, and larger payloads typically go into Amazon S3 with a reference in the message.
The lines are blurring
Each system now covers some of the others' ground. Kafka has share groups for queue-style consumption. RabbitMQ has streams for log-style replay, and its quorum queues now support priorities. SQS remains the purest queue of the three.
That does not make the choice irrelevant. It means that the core design of each system is still where it is strongest, and the newer features are best treated as useful extras rather than reasons to pick a tool against its grain.
Which one should you pick?
Several services need the same order events (billing, analytics, search, fraud detection), and you want to replay them: Kafka.
Background jobs on AWS, such as sending emails, resizing images, or processing uploads, with retries and a dead letter queue: SQS, often together with Lambda.
Complex routing by message type or region, request-reply, or devices speaking MQTT: RabbitMQ.
Real-time analytics, stream processing, or change data capture: Kafka.
Per-message priorities: RabbitMQ.
A small team with low volume and no appetite for operations: SQS, or a managed RabbitMQ.
On-premises or multi-cloud deployment: Kafka or RabbitMQ, since SQS is AWS-only.
A useful test is to ask one question: should each message be handled by exactly one worker and then forgotten, or should many readers be able to consume the same history? The first points to a queue (SQS or RabbitMQ), and the second points to a log (Kafka).
Mistakes to avoid
Using Kafka as a simple job queue at low volume. You take on a cluster's operational weight to solve a problem SQS or RabbitMQ would handle in an afternoon.
Expecting replay from a queue. Once a message is acknowledged or deleted in SQS or a RabbitMQ queue, it is gone.
Assuming SQS Standard keeps order or never duplicates. It does neither. Use FIFO if you need order, and still design for repeats.
Assuming FIFO is just Standard with ordering. FIFO queues have lower default throughput limits, so check them against your load.
Choosing by popularity. The most widely used tool is not necessarily the right one for your access pattern.
Final thoughts
These three tools solve neighboring problems, not the same one. Kafka keeps a durable history that many systems can share, RabbitMQ routes messages intelligently to the right workers, and SQS hands you a reliable queue without any servers. Start from how your messages will be used, who needs to read them, and whether anyone will ever need to read them again. The right choice usually becomes clear from there, and many mature systems end up using more than one.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!