Cloud Tech by Victor
System DesignIntermediate

The CAP Theorem

Why partition tolerance isn't actually optional for a distributed system, and why that leaves only a choice between consistency and availability once a real network partition happens, not a free choice among all three properties all the time.

Updated 2026-07-244 min read

Overview

CAP describes consistency (every read sees the latest write, everywhere), availability (the system keeps responding to requests), and partition tolerance (the system keeps operating even when nodes can't communicate). The common "choose two of three" framing is slightly misleading: partition tolerance isn't a real, standing choice, network partitions happen regardless of what a system "picks," so the actual decision CAP describes only activates during a real partition, and it's a choice between consistency and availability specifically in that moment, not a constant, permanent trade-off a system lives with at all times. A CP system pauses or rejects requests during a partition to guarantee correctness (a bank refusing a transfer rather than risking a duplicate); an AP system keeps responding and reconciles afterward (a chat app that stays usable through a network split). Systems like MongoDB make this a concrete, tunable choice via write concern and read preference, not an abstract, fixed architectural decision.

Quick Reference

PropertyMeansNon-negotiable?
ConsistencyEvery read reflects the latest writeNo, tradeable during a partition
AvailabilitySystem keeps responding to requestsNo, tradeable during a partition
Partition toleranceSystem keeps operating despite node communication failuresEffectively yes, for any real distributed system
During an actual partitionCP behaviorAP behavior
Pauses/rejects some requestsYes, to stay consistentNo
Keeps respondingNoYes, may return stale data
FitsFinancial transactions, inventory countsChat, social feeds, caching

Syntax

javascript
// MongoDB - w: 'majority' waits for the write to be acknowledged
// (and durable) on a majority of replicas before returning.
db.collection.insertOne(doc, { writeConcern: { w: 'majority' } });

// w: 1 only waits for acknowledgment from the primary, a weaker
// durability guarantee in exchange for a faster write.
db.collection.insertOne(doc, { writeConcern: { w: 1 } });

Examples

javascript
// Reading from a secondary favors availability over strict
// consistency; the data returned might lag the primary. Whether a
// read reliably sees a prior write depends on readConcern, session
// (causal consistency), and readPref together, not readPref alone.
db.collection.find().readPref('secondary');

CAP is about behavior during a partition, not a permanent state

A well-built system can be both consistent and available when nothing is broken. CAP describes what has to give specifically once nodes actually can't communicate, don't treat it as a constant, all-the-time architectural trade-off.

Visual Diagram

Common Mistakes

  • Treating "choose two of three" as a constant, standing architectural decision, rather than a choice that only actually activates during a real network partition.
  • Assuming partition tolerance is optional for a distributed system, when network partitions are a fact of distributed operation, not an avoidable scenario.
  • Picking CP or AP for an entire system uniformly, when different operations within the same system (a payment versus a "last seen" timestamp) often warrant different choices.
  • Confusing CAP's consistency (every read sees the latest write) with ACID's consistency (a transaction moves the database from one valid state to another), they are different, related-sounding concepts.

Performance

  • CP behavior during a partition trades availability (rejected/paused requests) for correctness; the "cost" is measured in failed or delayed requests during the partition window, not steady-state latency.
  • AP behavior keeps latency and availability high during a partition at the cost of temporary inconsistency that has to be reconciled afterward, real reconciliation work, not a free lunch.

Best Practices

  • Decide CP versus AP per operation based on what a wrong answer actually costs, financial correctness usually demands CP, a social feed usually tolerates AP.
  • Use tunable consistency settings (write concern, read preference, quorum size) where the database supports them, rather than treating CP/AP as a fixed, whole-system choice.
  • Design explicitly for the reconciliation step an AP choice implies, conflicting writes made during a partition don't resolve themselves.
  • Don't conflate CAP's "consistency" with ACID's; know which one a given conversation or requirement is actually about.

Interview questions

Why is the CAP theorem often described as "choose two of three," and why is that framing slightly misleading?

The framing suggests a system designer picks any two of consistency, availability, and partition tolerance as a free, standing choice, but partition tolerance isn't actually optional for a real distributed system, network partitions happen (a link fails, a node becomes unreachable), so a system that "chooses" not to tolerate partitions simply isn't distributed in any meaningful sense once one occurs. The real choice CAP describes is narrower and only activates during an actual partition: when nodes can't communicate, do you keep responding and risk inconsistency (AP), or do you pause and refuse some requests to guarantee consistency (CP)? Outside of an actual partition, a well-designed system can be both consistent and available; CAP is a statement about what happens specifically during a partition, not a permanent, constant trade-off.

A banking system typically favors CP behavior during a partition, while a chat application typically favors AP. What does each system actually do differently when a partition occurs, and why does the choice fit each use case?

A CP system, during a partition, pauses or rejects requests that can't be guaranteed consistent, a bank stopping a transfer rather than risking two nodes independently approving withdrawals against the same balance, since a duplicated or lost transaction is a correctness failure worse than a temporary outage. An AP system keeps responding during the partition, accepting the risk of temporarily inconsistent state, a chat app still accepting and displaying messages on both sides of a network split, reconciling them once the partition heals, because staying available and eventually consistent matters more to users than every message reappearing everywhere in a strict, immediate order.

MongoDB can be configured to behave more like a CP system or more like an AP system. What settings make that choice, and what are they actually trading?

Write concern and read preference are the actual levers: a write concern requiring acknowledgment from a majority of replicas favors consistency, a write isn't considered successful until enough nodes agree, at the cost of availability if too many nodes are unreachable during a partition. A looser write concern, or reading from secondaries that might lag behind the primary, favors availability, operations keep succeeding even when full replica agreement isn't achievable, at the cost of potentially reading or acknowledging data that isn't fully consistent across the cluster yet. This is exactly the CP-versus-AP trade-off CAP describes, expressed as a concrete, tunable configuration rather than an abstract theorem.

References

ShareXLinkedInReddit
Was this page helpful?
Suggest an improvement