---
title: "Event-Driven Architecture Without the Hype: When Queues and Pub/Sub Help, and When They Hurt"
author: "Alysaa Lissy"
published: "2026-11-12T05:17:58.904Z"
updated: "2026-11-12T05:17:58.904Z"
url: "https://zyvop.com/event-driven-architecture-without-the-hype-when-queues-and-pub-sub-help-and-when-they-hurt-vw8xs"
canonical: "https://zyvop.com/event-driven-architecture-without-the-hype-when-queues-and-pub-sub-help-and-when-they-hurt-vw8xs"
tags: ["Messaging","pub-sub","queues","event-driven","Architecture","Distributed Systems"]
---

# Event-Driven Architecture Without the Hype: When Queues and Pub/Sub Help, and When They Hurt

An honest guide to event-driven architecture: when queues, pub/sub and logs earn their complexity, when a plain function call wins, and the patterns that keep events from biting you.

![Cover image](<https://pub-03c883138afa4f88bc40a259a9109c50.r2.dev/posts/covers/event-driven-architecture-without-the-hype-when-queues-and-p-xeeaxs2o.webp>)

> **TL;DR**
> 
> -   An **event** is a record that something happened. A broker lets the sender stop waiting for whoever cares.
>     
> -   Events earn their keep when they take **slow, optional or fan-out work** off the critical path, absorb **spikes**, and let teams add consumers **without touching the producer**.
>     
> -   They are the wrong tool when the caller **needs the answer**, when an **invariant must hold at commit time**, or when a plain function call would do.
>     
> -   The price is **duplicates, reordering, eventual consistency and flows no single file describes**. Pay it with idempotent consumers, a transactional outbox, dead-letter queues, lag alerts and trace IDs.
>     

Event-driven architecture has the best sales pitch in software: decoupling, scalability and resilience. All three are real. All three also come with invoices that the pitch deck forgot to mention.

This post is about reading those invoices before you sign. We will start with a system that genuinely has a problem, see exactly what events fix, then spend most of our time on what they break and how to defend against it.

* * *

## 1\. Start with a real problem: a checkout that does too much

Imagine a checkout API that does everything inline, one call after another:

```mermaid
flowchart LR
    C["Client"] --> API["Checkout API"]
    API --> INV["Inventory<br/>40 ms"]
    INV --> PAY["Payments<br/>300 ms"]
    PAY --> MAIL["Email service<br/>250 ms"]
    MAIL --> LOY["Loyalty points<br/>60 ms"]
    LOY --> ANA["Analytics<br/>30 ms"]
    ANA --> WH["Warehouse<br/>120 ms"]
    WH --> OK["201 Created<br/>after about 800 ms"]
```

This design has three separate problems, and they are worth naming because events fix each one differently:

1.  **Latency adds up.** The user waits for the sum: 40 + 300 + 250 + 60 + 30 + 120 = **800 ms**. Most of that is work the buyer does not care about.
    
2.  **Availability multiplies.** If every service is up 99.9% of the time and a request needs all six, the chain is up about 0.999⁶ = **99.4%**. Over a 30-day month, a single 99.9% dependency is down about 43 minutes. The chain of six is down about **259 minutes** (4.3 hours). That assumes independent failures, every call required, and no retries.
    
3.  **Coupling is in the code.** Every time someone wants a new reaction to a purchase (a fraud check, a referral reward), the checkout team has to change and redeploy the checkout.
    

Notice what is _not_ a problem here: nothing is slow because of a lack of architecture. The email service being down should not stop someone buying a book, but that is a question about **which calls belong on the critical path**, and we will answer it deliberately rather than by turning everything into an event.

* * *

## 2\. What people actually mean by "event-driven"

An **event** is an immutable fact about the past: `order.placed`. A **command** is a request for the future: `send_email`. Events are named in the past tense, and the sender does not know or care who listens. That is the whole trick: the dependency is reversed, so the producer no longer has to know its consumers.

But "event-driven" covers several quite different designs. Martin Fowler's well-known article separates them, and the distinction saves a lot of confusion:

| Pattern | What the event carries | What the consumer does | Trade-off |
| --- | --- | --- | --- |
| **Event notification** | Almost nothing: an id and a type | Calls the producer back for details | Simple and small, but you create hidden call-back coupling and lose sight of the overall flow |
| **Event-carried state transfer** | The data that changed | Keeps its own copy and never calls back | The consumer keeps working when the producer is down, at the cost of duplicated data and eventual consistency |
| **Event sourcing** | Every state change, as the source of truth | Rebuilds state by replaying the log | Powerful audit and replay, but a different persistence model altogether |

Most systems need the first two. **Event sourcing is a separate, heavy decision**, and CQRS (separate read and write models) is a separate decision again that does not even require events. Adopting a message broker commits you to none of them.

### Three delivery shapes

Brokers come in three shapes, and picking the right shape matters more than picking a product:

```mermaid
flowchart LR
    subgraph L["Log: retained and replayable, one offset per group"]
        P3["Producer"] --> LL[["Partitioned log"]]
        LL -->|"group A"| G1["Consumers share the partitions"]
        LL -->|"group B, own offset"| G2["Consumer replays from the start"]
    end
    subgraph T["Pub/sub topic: every subscriber gets a COPY"]
        P2["Publisher"] --> TT[["Topic"]]
        TT --> S1["Email service"]
        TT --> S2["Loyalty service"]
    end
    subgraph Q["Work queue: each message goes to ONE worker"]
        P1["Producer"] --> QQ[["Queue"]]
        QQ --> W1["Worker 1"]
        QQ --> W2["Worker 2"]
    end
```

-   A **work queue** spreads tasks across competing workers. Once a message is acknowledged, it is gone.
    
-   A **pub/sub topic** broadcasts: every subscriber gets its own copy. (Plain SQS, for example, does not fan out by itself. The usual AWS pattern is a topic with one queue subscribed per consumer.)
    
-   A **log** (Kafka is the classic example) keeps records for a retention period whether or not they were consumed, and each consumer group tracks its own position. Kafka's _consumer group_ generalizes the other two shapes: consumers sharing a group split the work like a queue, while separate groups each see everything like pub/sub. Because position belongs to the consumer, it can rewind and reprocess old data.
    

* * *

## 3\. What events genuinely buy you

Strip away the marketing and there are five real benefits:

1.  **Take slow or optional work off the critical path.** The buyer should not wait for the confirmation email to be sent.
    
2.  **Fan out to independent consumers.** Add the fraud check next quarter without editing checkout.
    
3.  **Absorb spikes.** A queue is a shock absorber between a bursty producer and a steady consumer. (A shock absorber, not a magic wand: it _defers_ work, it does not remove it. See section 5.5.)
    
4.  **Isolate failures.** If loyalty points are down, checkout still works and the points are granted when the service recovers.
    
5.  **Keep a history.** A durable log lets you replay, audit and build new views of old data.
    

Here is the checkout again, redesigned with those benefits and nothing more:

```mermaid
sequenceDiagram
    autonumber
    participant C as Client
    participant API as Checkout API
    participant DB as Orders DB
    participant R as Outbox relay
    participant B as Broker
    participant E as Email service
    participant L as Loyalty service

    C->>API: POST /checkout
    Note over API: Reserve stock and charge the card synchronously, because the user needs those answers
    API->>DB: BEGIN, INSERT order, INSERT outbox row, COMMIT
    API-->>C: 201 Created (order id)
    Note over C,API: The user is done. Everything below happens in the background.
    R->>DB: poll for unpublished outbox rows
    R->>B: publish order.placed
    R->>DB: mark the row published
    par fan-out
        B->>E: deliver order.placed
        E->>E: send confirmation email
    and
        B->>L: deliver order.placed
        L->>L: add loyalty points
    end
    E-->>B: ack
    L-->>B: ack
```

The most important thing in this diagram is what we **did not** turn into an event. Reserving stock protects an invariant (never sell what you do not have), and charging the card produces an answer the user is waiting for. Both stay as synchronous calls. Only the optional reactions (email, loyalty, analytics, warehouse notification) moved off the critical path.

The results, using the same illustrative numbers: latency drops from 800 ms to about **340 ms** (40 + 300, plus a few milliseconds for the database write). Required dependencies drop from six to two, so availability improves from about 99.4% to about **99.8%** (0.999²), or roughly 86 minutes of downtime per month instead of 259.

(The _outbox relay_ in the diagram is the piece that makes publishing reliable. It gets its own explanation in section 5.2.)

* * *

## 4\. Should this be an event? A decision flow

Run each interaction through these questions before you reach for a broker:

```mermaid
flowchart TD
    A["Service A needs something done by service B"] --> B{"Does A need the result to continue?"}
    B -->|Yes| S1["Call it synchronously<br/>(HTTP or gRPC)"]
    B -->|No| C{"Must a business invariant hold at commit time?<br/>(no overselling, balance never negative)"}
    C -->|Yes| S2["Keep it in one transaction<br/>or call it synchronously"]
    C -->|No| D{"How many independent parties care?"}
    D -->|"One, and it can wait"| Q1["Work queue or background job"]
    D -->|"Several, now or later"| P1["Pub/sub topic"]
    D -->|"Several, and I need history or replay"| L1["Durable log"]
```

If you answer "yes" to either of the first two questions, an event is probably the wrong tool, and the sections below explain why.

* * *

## 5\. What you sign up for

Every benefit above has a matching failure mode. These seven are the ones that actually page people.

### 5.1 Duplicates: delivery is at-least-once

Mainstream brokers choose to deliver a message _at least once_ rather than risk losing it. Amazon's SQS documentation says this plainly for standard queues: a message may be delivered more than once, and you should design for it with idempotent operations. Even SQS FIFO queues, which deduplicate _sends_, cannot stop your handler from running twice if it crashes after doing the work and before acknowledging the message.

```mermaid
sequenceDiagram
    participant B as Broker
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant D as Database

    B->>W1: deliver event e1 (add 500 points)
    W1->>D: apply the change (and record e1 if idempotent)
    D-->>W1: committed
    Note over W1: Crashes before sending the ack
    Note over B: No ack, so the visibility timeout expires
    B->>W2: redeliver event e1
    alt consumer is NOT idempotent
        W2->>D: UPDATE points = points + 500
        Note over D: Customer got 1000 points. Duplicate side effect.
    else consumer is idempotent
        W2->>D: INSERT INTO processed_events (e1)
        D-->>W2: conflict, already processed
        W2-->>B: ack and do nothing
    end
```

You cannot prevent this from happening. You can only make it harmless. The standard defense is an **idempotent consumer**: record each processed event id in the **same transaction** as the side effect, and skip events you have already seen.

```javascript
async function handleOrderPlaced(event) {
  await db.tx(async (tx) => {
    // 1. Claim the event id. If we have seen it before, this inserts nothing.
    const { rowCount } = await tx.query(
      'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING',
      [event.id]
    );
    if (rowCount === 0) return;                     // duplicate delivery: do nothing

    // 2. The side effect, in the SAME transaction as the claim.
    const points = Math.floor(event.data.totalCents / 100);
    await tx.query(
      'UPDATE accounts SET points = points + $1 WHERE id = $2',
      [points, event.data.customerId]
    );
  });                                               // then ack the message
}
```

The "same transaction" part is the whole point. If the side effect fails and rolls back, the claim rolls back with it, so the redelivered message gets a fresh chance instead of being wrongly skipped as "already processed."

For side effects _outside_ your database (sending an email, charging a card), a database transaction cannot cover them. Pass the event id as an **idempotency key** to the provider where it supports one (many payment APIs do), or record a "sent" state before and after the call and accept a rare double send.

> "Exactly-once" guarantees are usually scoped to the broker's own boundary. The moment your handler touches an outside system (a database, an email provider), you are back to **at-least-once delivery plus an idempotent consumer**.

### 5.2 The dual-write problem, and the outbox

Look at the obvious way to publish an event after saving data:

```javascript
// BROKEN: two writes to two systems, and no transaction spans both
async function placeOrderBroken(order) {
  await db.query(
    'INSERT INTO orders (id, customer_id, total_cents, status) VALUES ($1, $2, $3, $4)',
    [order.id, order.customerId, order.totalCents, 'placed']
  );
  await broker.publish('order.placed', { orderId: order.id });  // crash or timeout here...
}
```

```mermaid
sequenceDiagram
    participant API as Checkout API
    participant DB as Database
    participant B as Broker

    Note over API,B: Dual write: two systems and no shared transaction
    API->>DB: COMMIT order 42
    DB-->>API: ok
    API-xB: publish order.placed (crash or network error)
    Note over DB,B: The order exists but the event is lost. No email, no points, no shipment.
```

If the process dies between the two lines, the order exists and **the event never will**. Swap the order of the two lines and you get the mirror-image bug: publish first, then the database write fails, and consumers react to an order that does not exist (a "ghost" event). In-process retries cannot save you in either ordering, because the failure can be the process itself dying.

The fix is the **transactional outbox** (described by Chris Richardson). Stop writing to two systems. Write the event into an `outbox` table **in the same database transaction** as the business change, and let a separate relay publish it afterwards:

```sql
CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,
  event_id     TEXT NOT NULL UNIQUE,
  type         TEXT NOT NULL,
  payload      JSONB NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);
```

```javascript
async function placeOrder(order) {
  const event = {
    id: crypto.randomUUID(),
    type: 'order.placed',
    schemaVersion: 1,
    sequence: 1,
    occurredAt: new Date().toISOString(),
    data: { orderId: order.id, customerId: order.customerId, totalCents: order.totalCents },
  };

  await db.tx(async (tx) => {
    await tx.query(
      'INSERT INTO orders (id, customer_id, total_cents, status) VALUES ($1, $2, $3, $4)',
      [order.id, order.customerId, order.totalCents, 'placed']
    );
    await tx.query(
      'INSERT INTO outbox (event_id, type, payload) VALUES ($1, $2, $3)',
      [event.id, event.type, JSON.stringify(event)]
    );
  });                                    // both rows commit together, or neither does
  return order.id;
}
```

```javascript
async function relayOnce() {
  const { rows } = await db.query(
    'SELECT id, event_id, type, payload FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100'
  );
  for (const row of rows) {
    await broker.publish(row.type, row.payload, { messageId: row.event_id });  // may throw: stop here
    await db.query('UPDATE outbox SET published_at = now() WHERE id = $1', [row.id]);
  }
}
```

Three details make this work in production:

-   **The relay is at-least-once, too.** If it crashes after publishing and before marking the row, it publishes that row again on restart. That is fine, because your consumers are already idempotent (section 5.1). The two patterns are a pair.
    
-   **Order is preserved by stopping on failure.** If `publish` throws, the loop aborts, so later events cannot overtake the failed one.
    
-   **Polling is the simple relay; change data capture (CDC) is the efficient one.** Polling adds query load and latency. Tailing the database's transaction log (tools like Debezium do this for Postgres) is faster but ties you to the database's replication protocol. To run several relay instances, use row locking such as Postgres `FOR UPDATE SKIP LOCKED`, but know that concurrent relays give up strict publish order. If order matters, run a single relay.
    

Monitor the **outbox depth and the age of its oldest unpublished row**. A growing outbox means the relay or the broker is stuck.

### 5.3 Ordering: you get less than you think

Brokers order messages only within a narrow scope:

-   **Kafka** gives a total order only _within a partition_, not across the partitions of a topic. The producer chooses the partition, usually by key.
    
-   **SQS standard queues** give best-effort ordering only. **SQS FIFO queues** order strictly _within a message group_.
    

The practical rule is to **choose the key to be the entity id** (the order id, the customer id), so all events about one entity travel through one ordered lane. Never assume a global order.

Even then, retries and redeliveries can make an old event arrive after a newer one. Defend with a **per-entity sequence number** and ignore anything stale:

```javascript
async function applyOrderStatus(event) {
  // order_view(id, status, version): a read model built from events
  const { rowCount } = await db.query(
    `UPDATE order_view SET status = $1, version = $2
      WHERE id = $3 AND version < $2`,
    [event.data.status, event.sequence, event.data.orderId]
  );
  return rowCount === 1;   // false means the event was stale or a duplicate: ignore it
}
```

This guard also makes the handler idempotent for free: a duplicate carries the same sequence number, so `version < $2` is false the second time.

### 5.4 Poison messages and dead-letter queues

Some messages will never succeed: malformed data, a bug, a record that violates a constraint. Retried forever, one poison message wastes capacity (and in an ordered lane, **blocks everything behind it**). Give every message a bounded life:

```mermaid
stateDiagram-v2
    [*] --> Ready: published
    Ready --> InFlight: delivered to a consumer
    InFlight --> Done: consumer acks
    InFlight --> Ready: ack timeout or nack (retry, ideally with backoff)
    InFlight --> DeadLetter: max attempts exceeded
    DeadLetter --> Ready: redrive after the fix
    Done --> [*]
```

In SQS terms, the _visibility timeout_ hides a delivered message from other consumers (30 seconds by default), and a _redrive policy_ with `maxReceiveCount` moves a message to a **dead-letter queue (DLQ)** after that many failed receives. The queue only helps if a human is told: **alert when the DLQ is non-empty**, and keep a runbook for inspecting messages and redriving them after a fix.

Set the visibility timeout _longer than your handler's worst-case run time_. If it is shorter, a healthy but slow handler gets its message redelivered to a second worker, and you have manufactured a duplicate.

### 5.5 Backlogs: queues defer problems, they do not solve them

A queue hides overload until it becomes a crisis. If messages arrive faster than consumers drain them, the backlog grows without bound. Say 500 messages per second arrive but your consumers manage only 400: the queue gains 100 messages every second, **6,000 per minute**. After ten minutes you are 60,000 messages behind, and users are still being told that "the email is on its way."

Scaling consumers to 600 per second only drains the backlog at the _surplus_ rate, 100 per second, so recovery takes as long as the build-up did:

```text
Messages waiting (1 block = 2,000 messages)

min  0 | 0
min  2 | ██████ 12,000
min  4 | ████████████ 24,000
min  6 | ██████████████████ 36,000
min  8 | ████████████████████████ 48,000
min 10 | ██████████████████████████████ 60,000  <- consumers scaled up
min 12 | ████████████████████████ 48,000
min 14 | ██████████████████ 36,000
min 16 | ████████████ 24,000
min 18 | ██████ 12,000
min 20 | 0  <- backlog cleared
min 22 | 0
min 24 | 0
```

Alert on the **age of the oldest message** (or consumer lag in a log-based system), not just queue depth. Age answers the question users actually care about: "how late is my email?" Also remember that queues have **retention limits**, and messages older than the limit are discarded.

### 5.6 Eventual consistency: where did my order go?

Anything derived from events lags behind the source of truth. In our checkout the order row is committed _before_ the response, so "My orders" works immediately. What lags is the **derived** data: the loyalty balance, the warehouse status, an analytics dashboard.

Be explicit about which data is which. For anything the user will look at right after acting, either read it from the primary store (read-your-writes), return the created resource in the response, or show an honest "processing" state. A user who refreshes and sees nothing will resubmit.

### 5.7 Debugging flows that no file describes

With a synchronous call, a stack trace tells you what happened. With events, the cause and the effect are in different processes, possibly minutes apart. Fowler warns that it is easy to build nicely decoupled systems with event notification, lose sight of the larger-scale flow, and pay for it years later.

The defense is cheap but must be built in from day one:

-   Put a **correlation id** on every event and log it on every line.
    
-   Propagate **trace context** from producer to consumer in the message headers. The W3C Trace Context standard defines the `traceparent` header for exactly this, for example `traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01` (version, trace id, parent id, flags).
    
-   Keep a **catalogue of events**: who produces each one, and who consumes it.
    

* * *

## 6\. Choreography vs. orchestration

When several services must cooperate on one business process, you have two ways to wire them:

```mermaid
flowchart LR
    subgraph OR["Orchestration: one coordinator owns the flow"]
        direction LR
        W["Order workflow"] -->|"1. reserve"| I2["Inventory"]
        W -->|"2. charge"| P2["Payments"]
        W -->|"3. ship"| S2["Shipping"]
        W -.->|"on failure: compensate"| I2
    end
    subgraph CH["Choreography: services react to each other's events"]
        direction LR
        O1["Orders"] -->|"order.placed"| I1["Inventory"]
        I1 -->|"stock.reserved"| P1["Payments"]
        P1 -->|"payment.captured"| S1["Shipping"]
        P1 -->|"payment.failed"| I1
    end
```

**Choreography** has no central owner. It is lovely for simple fan-out ("when an order is placed, several teams react independently"). But for a multi-step process, the logic is smeared across services: to learn what happens when a payment fails, you have to read four codebases, and cycles (like `payment.failed` flowing back to Inventory) are easy to create by accident.

**Orchestration** puts the flow in one place: a workflow or saga coordinator calls each step and runs **compensating actions** (release the stock, refund the card) when a later step fails. You get one place to read the flow, one place to see its current state, and one place to add a timeout.

A sound default: **choreography for independent reactions, orchestration for business processes with ordering and undo.**

* * *

## 7\. When not to use events

| Situation | Why events hurt | Better choice |
| --- | --- | --- |
| The caller needs the answer to proceed (price quote, login, card authorization) | Request-reply over a queue reinvents HTTP with extra latency and extra ways to fail | A synchronous call |
| An invariant must hold at commit time (do not oversell, usernames are unique) | Eventual consistency lets two concurrent requests both "win" | One transaction, or a unique constraint |
| One service, one team, one database | A broker adds infrastructure and failure modes with no decoupling to show for it | A function call, or a simple background job |
| Low volume and a simple flow | Operational overhead exceeds the benefit | A cron job or a database-backed job table |
| You want to avoid designing an API | Events become an untyped, undocumented API, and you get a _distributed monolith_ | Design explicit contracts first |
| Nobody owns monitoring | The DLQ fills up silently and lag grows unnoticed | Do not ship until the alerts exist |

* * *

## 8\. Designing a good event

An event is a public API with an unknown number of future consumers, so design it like one:

```json
{
  "id": "0b1f6c9e-3f0a-4c55-9a1e-6f2d6f1c7a10",
  "type": "order.placed",
  "schemaVersion": 1,
  "sequence": 1,
  "occurredAt": "2026-11-12T09:30:00Z",
  "correlationId": "req_7f3a9c1e",
  "data": {
    "orderId": "ord_42",
    "customerId": "cus_9",
    "totalCents": 4999,
    "currency": "USD"
  }
}
```

1.  **Name it as a past-tense fact**: `order.placed`, not `send.email`. Commands couple you to a specific consumer.
    
2.  **Give it a unique** `id`**.** It is the deduplication key for everything in section 5.1.
    
3.  **Version the schema, and evolve it additively.** Add optional fields. Never change the meaning of an existing one.
    
4.  **Include a per-entity** `sequence` when order matters (section 5.3).
    
5.  **Carry correlation and trace context** (section 5.7).
    
6.  **Use the entity id as the partition or group key** so one entity's events stay ordered.
    
7.  **Choose thin vs. fat on purpose.** A fat event (carrying the data) lets consumers work when the producer is down but spreads copies of the data around. A thin one is smaller but forces callbacks. Keep secrets and personal data out, and pass large blobs by reference.
    

* * *

## 9\. Picking a broker without the hype

Choose the _shape_ first, then the product. The names below are examples, not a ranking, and many products can play more than one role:

| You need | Shape | Examples |
| --- | --- | --- |
| Background jobs: one worker per task | Work queue | SQS, RabbitMQ, Redis-based job libraries |
| Fan-out to independent consumers | Pub/sub topic, often with one queue per consumer | SNS plus SQS, Google Pub/Sub, NATS |
| Replay, history, very high throughput, stream processing | Partitioned log | Kafka and Kafka-compatible systems, Kinesis |

And remember: for modest volumes, many teams run job queues on the database they already operate. It is boring, it is transactional with your data, and it deserves a seat at the table before you add a new piece of infrastructure.

* * *

## 10\. Adopting events incrementally

You do not have to "go event-driven." Walk up the ladder only as far as your problems demand:

1.  **Move one slow, optional side effect** (the confirmation email) into a background job. Measure what changes.
    
2.  **Add the outbox**, so publishing is atomic with your data change, and make the consumer idempotent.
    
3.  **Add a second consumer** (loyalty points). You now have pub/sub, and that is when decoupling starts paying off.
    
4.  **Before the third consumer:** dead-letter queue with an alert, age-of-oldest-message alert, trace propagation, and a written event catalogue.
    
5.  **Reach for a replayable log** only when you have a concrete need for replay, audit, or stream processing.
    

### The day-2 checklist

-   Every consumer is idempotent, with a tested duplicate case.
    
-   Events are published through an outbox (or an equivalent atomic mechanism).
    
-   Every queue has a DLQ, a bounded retry policy and an alert on DLQ depth.
    
-   You alert on the **age** of the oldest message or on consumer lag.
    
-   Trace context and a correlation id travel with every event.
    
-   Event schemas are versioned and evolve additively.
    
-   There is a runbook for redrive and replay, and a named owner for each event.
    

* * *

## 11\. The mental model on one screen

1.  An **event** is a fact about the past. The producer does not know its consumers, and that is both the benefit and the risk.
    
2.  Events fix **latency, availability and coupling** on the critical path, but only for work the caller does not need the answer to.
    
3.  Keep **answers and invariants synchronous**. Move **optional reactions** to events.
    
4.  Delivery is **at-least-once**. Make consumers **idempotent**, and publish with an **outbox**.
    
5.  **Order is scoped** (per partition or message group), so key by entity and guard with sequence numbers.
    
6.  A queue **defers** work. Watch the **age of the oldest message**, and keep a **DLQ** with an alert.
    
7.  Pay for **observability** (correlation ids, trace context, an event catalogue) on day one, not after the first incident.
    

* * *

## Further reading

-   [What do you mean by "Event-Driven"?](<https://www.martinfowler.com/articles/201701-event-driven.html>), Martin Fowler. The article that untangles event notification, event-carried state transfer, event sourcing and CQRS.
    
-   [Pattern: Transactional outbox](<https://microservices.io/patterns/data/transactional-outbox.html>), Chris Richardson. The dual-write problem and both relay styles (polling and log tailing).
    
-   [Amazon SQS queue types](<https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-types.html>). Delivery guarantees, ordering and deduplication for standard and FIFO queues, straight from the documentation.
    
-   [Apache Kafka documentation](<https://kafka.apache.org/documentation/>). Topics, partitions, consumer groups, offsets and retention.
    
-   [W3C Trace Context](<https://www.w3.org/TR/trace-context/>). The standard `traceparent` and `tracestate` headers for propagating trace identity across services.
    
-   _Designing Data-Intensive Applications_ by Martin Kleppmann, chapter 11 (stream processing), for a rigorous treatment of dual writes and log-based integration.
    

If this post helped, send it to the teammate who is about to put a queue between two functions in the same process.