System Design

CAP Theorem and PACELC: What Distributed Systems Actually Trade Away

Why distributed databases can't have everything, and how PACELC explains the trade-offs you live with even when the network is healthy.

·
8 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

CAP Theorem and PACELC: What Distributed Systems Actually Trade Away

If you've read anything about distributed databases, you've seen the CAP theorem. It usually shows up as a triangle with the advice "pick two." That advice is catchy, but it's misleading, and it leaves out half of the story.

This post covers what CAP really says, where people get it wrong, and how PACELC fills the gap. That way you can reason about the trade-offs you deal with every day, not just during outages.

If you build on top of databases rather than build them, the practical part is further down: the settings, code, and failure handling that move you along the trade-off.

The three letters in CAP

Eric Brewer proposed CAP as a conjecture in 2000, and Seth Gilbert and Nancy Lynch proved a formal version in 2002. The three properties have narrower meanings than the words suggest:

  • Consistency (C): every read sees the most recent write, or returns an error. Formally this is linearizability: the system behaves as if there were a single copy of the data.

  • Availability (A): every request sent to a non-failing node gets a non-error response. There's no promise that the response contains the latest data.

  • Partition tolerance (P): the system keeps operating even when the network drops or delays messages between nodes.

Why it's really a choice between C and A

In any system that spans more than one machine, network partitions will happen. Cables fail, switches misbehave, a cloud zone gets isolated.

You don't get to opt out of P, so the useful question isn't "which two do I pick?" It's "when a partition happens, what do I give up?"

Here's the smallest example. Two nodes hold copies of an account balance, and the link between them breaks:

Client 1 --write balance=50--> [ Node A ]   X   [ Node B ] <--read balance?-- Client 2
                                  (network partition)

Node B has no way of knowing about the write. It has two options:

  1. Refuse or time out until it can reach Node A again. The system stays consistent but gives up availability. This is a CP choice.

  2. Answer with what it has, which may be stale. The system stays available but gives up consistency. This is an AP choice.

There's no third option. That's the theorem.

CP and AP in the real world

Leans CP

Leans AP

etcd, ZooKeeper, Consul (coordination and config)

Cassandra, Riak (Dynamo-style stores)

HBase

DynamoDB (eventually consistent reads by default)

Spanner (CP, though highly available in practice)

CouchDB

Treat this table as a tendency, not a label. Many systems let you tune behavior per query, and a database's default settings don't always match how it's deployed.

Five common misunderstandings

1. "Pick any two of three." You can't really pick CA in a distributed system, because partitions aren't optional. A single-node database is "CA" only because there's no network between nodes to partition.

2. The C in CAP is not the C in ACID. ACID consistency means a transaction moves the database between valid states and respects constraints. CAP consistency is about replicas agreeing, so that reads reflect the latest write.

3. CAP availability is stricter than uptime. A system that's up 99.99% of the time can still be "unavailable" in the CAP sense if some healthy nodes refuse requests during a partition.

4. It isn't a permanent label on a system. A single database can behave CP for one operation and AP for another. Think per operation, not per product.

5. CAP only talks about partitions. Partitions are rare. Most of the time your network is fine, and CAP is silent about what you're trading then. That gap is exactly where PACELC comes in.

PACELC: the rest of the story

In 2010, Daniel Abadi proposed PACELC, later detailed in a 2012 paper in IEEE Computer. It's read like this:

If there's a Partition, choose between Availability and Consistency. Else, choose between Latency and Consistency.

The first half is just CAP. The second half is new, and it's the part you feel daily.

Why does latency enter the picture? Because of replication. To guarantee that a read sees the latest write, a write has to be acknowledged by enough replicas before you tell the client it succeeded.

If those replicas live in different regions, that means waiting on cross-region round trips, which often take tens to a couple hundred milliseconds depending on distance.

You can skip the wait by acknowledging after one replica and replicating in the background. Writes get fast, but a read from another replica might briefly see old data.

So even on a perfectly healthy network, you're choosing between speed and freshness.

The four combinations

Class

During a partition

Normal operation

Typical examples

PA/EL

Availability

Low latency

Cassandra, Riak, DynamoDB (default settings)

PC/EC

Consistency

Consistency

VoltDB, HBase, Spanner

PA/EC

Availability

Consistency

MongoDB (as classified in Abadi's paper)

PC/EL

Consistency

Low latency

PNUTS (Yahoo)

These classifications are approximations. Configuration changes them: a Cassandra cluster reading and writing at ALL behaves very differently from one using ONE. Use the table to build intuition, then check the documentation for the version and settings you actually run.

Turning the dial: quorums

Many Dynamo-style databases let you move along the PACELC spectrum per request using quorums. With N replicas, a write acknowledged by W of them, and a read that consults R of them:

  • If R + W > N, the read and write sets must overlap, so a read contacts at least one replica that has the latest acknowledged write.

  • With N = 3, W = 2, R = 2, reads are much fresher, at the cost of waiting on a second replica each time.

  • With W = 1, R = 1, you get low latency and high availability, but reads may be stale.

In Cassandra this shows up as consistency levels like ONE, QUORUM, LOCAL_QUORUM (a quorum within the local data center, which avoids cross-region waits), and ALL. It's the same trade-off, exposed as a setting you can change query by query.

One caveat worth knowing: overlapping quorums are not the same as linearizability. Concurrent writes, clock-based conflict resolution such as last-write-wins, and edge cases like sloppy quorums can still produce surprising reads.

If you need real compare-and-set behavior on a Dynamo-style store, use the feature built for it, such as Cassandra's lightweight transactions (IF NOT EXISTS and IF conditions). They run a consensus protocol, so expect noticeably higher latency.

What this looks like in code

For most developers, the CAP/PACELC trade-off shows up as a per-query option, not a theory question. Here are the same idea in three popular databases.

Cassandra (CQL, then the Node.js driver):

-- Fast, possibly stale
CONSISTENCY ONE;
SELECT balance FROM accounts WHERE id = 42;

-- Fresher, more latency
CONSISTENCY LOCAL_QUORUM;
SELECT balance FROM accounts WHERE id = 42;
import cassandra from "cassandra-driver";

await client.execute(
  "UPDATE accounts SET balance = ? WHERE id = ?",
  [50, 42],
  { prepare: true, consistency: cassandra.types.consistencies.localQuorum }
);

MongoDB (Node.js driver). Write concern and read concern control how many replicas must agree, and read preference controls where reads go. Reading from secondaries spreads load and is faster, but it can return stale data.

const accounts = db.collection("accounts", {
  writeConcern: { w: "majority" },
  readConcern: { level: "majority" },
});

await accounts.updateOne({ _id: 42 }, { $set: { balance: 50 } });

DynamoDB (AWS SDK v3). Reads are eventually consistent unless you ask otherwise:

import { DynamoDBClient, GetItemCommand } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({});
const { Item } = await client.send(
  new GetItemCommand({
    TableName: "Accounts",
    Key: { id: { S: "42" } },
    ConsistentRead: true, // default is false
  })
);

Strongly consistent reads in DynamoDB use more read capacity, can have higher latency, and aren't supported on global secondary indexes. That's PACELC showing up on your bill and your latency graph.

What your application still has to handle

Choosing a setting is the easy part. Your code has to live with the consequences.

  • Retries and idempotency. During a partition or slow replica, a timeout is ambiguous: the write may or may not have been applied. Use idempotency keys so a retry doesn't charge a customer twice.

  • Conflict resolution. With last-write-wins, two concurrent updates quietly become one. For counters, sets, or collaborative data, consider merge logic, CRDTs, or modeling changes as append-only events.

  • Read-your-writes. A user posts a comment, refreshes, and it's gone because the read hit a stale replica. Read from the primary or at a quorum right after a write, use sticky sessions, or use session guarantees like MongoDB's causally consistent sessions.

  • Failing well when you chose consistency. CP behavior means some requests will time out or error during a partition. Set sensible timeouts, return clear errors, and decide whether the feature can degrade to read-only instead of going fully dark.

Test the failure, not just the happy path

Most of this never shows up in local development, because your laptop has no partitions and near-zero latency.

Tools like Toxiproxy let you add latency, drop connections, and cut links between your app and its database in a test environment.

For a sense of how real databases behave under faults, the published Jepsen analyses are worth reading before you trust a vendor's consistency claims.

How to decide in practice

Ask one question about each piece of data: what's worse, a stale answer or a slow or failed one?

  • Favor consistency for money movement, inventory counts at checkout, unique usernames, distributed locks, and leader election. A wrong answer here causes double spends, oversold stock, or two nodes both believing they're the leader.

  • Favor availability and low latency for social feeds, like counts, view counters, recommendations, and session or shopping cart data. A slightly stale number is harmless, while a failed page load costs you users. Amazon's Dynamo paper made this case for shopping carts.

Most real systems mix both. An e-commerce platform might use a strongly consistent store for orders and payments and an eventually consistent one for product reviews and recommendations.

Using this in design reviews and interviews

When you're in an interview or a design review, avoid saying "we'll go with CP" and stopping there. Be specific instead:

  • Which operation are we talking about?

  • What failure are we designing for, a partition, a slow replica, a regional outage?

  • What should the user see when it happens: an error, a stale value, a retry?

  • What's the latency budget when everything is healthy?

Answering those four questions shows you understand the trade-off, not just the acronym.

Key takeaways

  • CAP says that during a network partition you must choose between consistency and availability. Partitions can't be avoided, so "pick two" is the wrong mental model.

  • CAP's consistency means linearizability, not ACID consistency, and its availability is stricter than everyday uptime.

  • PACELC extends CAP: when there's no partition, you still trade latency against consistency, mostly because of replication.

  • Real databases are configurable. Judge them by the settings you run, per operation, not by a one-word label.

  • Quorums reduce staleness but don't make a Dynamo-style store linearizable. Use conditional or transactional features when you need that.

  • In code, the trade-off lives in per-query settings (consistency level, read and write concern, ConsistentRead) and in how your app handles retries, conflicts, and stale reads.

  • Decide by asking what a stale answer costs compared to a slow or failed one.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Samod Alex

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Samod Alex's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.