Consistency vs Availability

In October 2012, a significant portion of AWS US-East went down. Services that prioritised consistency went offline entirely — they refused to serve requests rather than risk returning stale data. Services that prioritised availability kept running — they returned whatever data they had, even if it was minutes old. Neither choice was wrong. But every team that had not explicitly made a choice was caught off guard. This lesson is about making that choice deliberately, before the outage happens.

The CAP Theorem Revisited — From Theory to Production

You covered the CAP theorem in Lesson 5. The core claim: a distributed system can guarantee at most two of three properties — Consistency, Availability, and Partition tolerance. Since network partitions are unavoidable in any real distributed system, every database must choose between C and A when a partition occurs.

But the real world is messier than the theory. Systems are not permanently CP or permanently AP — most are tunable. Cassandra with QUORUM consistency behaves like a CP system. The same Cassandra cluster with ONE consistency behaves like an AP system. The decision is made per operation, not per database. The engineer who understands this is the one who sets consistency levels correctly for each use case instead of applying a single setting across everything.

Strong Consistency — Every Read Sees the Latest Write

Strong consistency means that after a write completes, every subsequent read from any node returns that write's value. There is a single, agreed-upon ordering of all operations. The database behaves as if it were a single machine regardless of how many nodes it has.

The cost is latency and availability. To achieve strong consistency in a multi-node system, writes must be coordinated across replicas before returning to the client. If a node is slow or unreachable, the write blocks or fails rather than proceeding without full consensus.

Eventual Consistency — Replicas Converge Over Time

Eventual consistency means that if no new writes occur, all replicas will eventually converge to the same value. There is no guarantee about how long "eventually" takes — it could be milliseconds or, in the event of a network partition, minutes. During that window, different nodes may return different values for the same key.

The benefit is speed and availability. Writes are acknowledged the moment one replica (or a small quorum) confirms them — the write does not wait for all replicas to sync. Reads can be served from any node regardless of whether it has the latest write. The system keeps running even during a partition.

Hands-on — Tuning Consistency in Cassandra

The scenario: You are the platform engineer for a fintech company running a 9-node Cassandra cluster with RF=3 across three data centres. The platform serves two very different workloads: a payment ledger that requires every read to see the latest committed balance, and a notification feed where a few seconds of staleness is fine. You need to configure the correct consistency levels for each and demonstrate the latency difference.

from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel
from cassandra.query import SimpleStatement
import time

session = Cluster(['10.0.1.1']).connect('fintech')

# --- PAYMENT LEDGER: strong consistency ---
# LOCAL_QUORUM: 2 of 3 replicas in local DC must agree
# W+R > N within DC ensures we always read the latest write
payment_read = SimpleStatement(
    "SELECT balance FROM accounts WHERE account_id = %s",
    consistency_level=ConsistencyLevel.LOCAL_QUORUM
)
payment_write = SimpleStatement(
    "UPDATE accounts SET balance = %s WHERE account_id = %s",
    consistency_level=ConsistencyLevel.LOCAL_QUORUM
)

t0 = time.time()
session.execute(payment_write, (4850.00, 'acc_441'))
row = session.execute(payment_read, ('acc_441',)).one()
payment_latency = (time.time() - t0) * 1000
print(f"Payment  balance: £{row.balance:.2f}  latency: {payment_latency:.1f}ms")

Payment  balance: £4850.00  latency: 6.2ms
Write acknowledged by 2/3 local replicas (LOCAL_QUORUM)
Read confirmed by 2/3 local replicas (LOCAL_QUORUM)
Guarantee: read always sees latest committed write ✓

# --- NOTIFICATION FEED: eventual consistency ---
# ONE: a single replica acknowledges the write and serves the read
# Fastest possible — acceptable for non-critical feed data
notif_read = SimpleStatement(
    "SELECT body, created_at FROM notifications WHERE user_id = %s LIMIT 20",
    consistency_level=ConsistencyLevel.ONE
)
notif_write = SimpleStatement(
    "INSERT INTO notifications (user_id, notif_id, body, created_at) "
    "VALUES (%s, uuid(), %s, toTimestamp(now()))",
    consistency_level=ConsistencyLevel.ONE
)

t0 = time.time()
session.execute(notif_write, ('usr_882', 'Alice liked your post'))
rows = session.execute(notif_read, ('usr_882',))
notif_latency = (time.time() - t0) * 1000
print(f"Notification feed latency: {notif_latency:.1f}ms  "
      f"rows: {len(list(rows))}")

Notification feed latency: 1.1ms  rows: 20
Write acknowledged by 1 replica (ONE)
Read served by nearest replica (ONE)
Trade-off: read may be up to ~200ms stale during replica lag ✓

Consistency Anomalies — The Failure Modes of Eventual Consistency

Eventual consistency is not a single guarantee — it is a spectrum. Depending on the system and the configuration, you may encounter different anomalies during the lag window between write and full replication.

The Consistency Decision Matrix

Behaviour During a Real Partition

The scenario: Your 6-node Cassandra cluster loses connectivity between its London and Frankfurt data centres. London nodes cannot reach Frankfurt nodes. Reads and writes are still arriving from both regions. You have configured different consistency levels for different data — and the partition reveals exactly what each setting means in practice.

from cassandra import Unavailable, WriteTimeout, ReadTimeout

def safe_payment_write(session, account_id, new_balance):
    """Payment write — strong consistency, will fail during partition."""
    stmt = SimpleStatement(
        "UPDATE accounts SET balance = %s WHERE account_id = %s",
        consistency_level=ConsistencyLevel.LOCAL_QUORUM
    )
    try:
        session.execute(stmt, (new_balance, account_id))
        return {"ok": True}
    except (Unavailable, WriteTimeout) as e:
        # LOCAL_QUORUM requires 2 of 3 replicas to ACK
        # If the partition leaves only 1 replica reachable: FAIL
        # Correct behaviour — refuse to write rather than risk inconsistency
        return {"ok": False, "reason": "quorum unavailable — partition detected",
                "action": "retry_after_partition_heals"}

def safe_feed_write(session, user_id, body):
    """Notification write — eventual consistency, survives partition."""
    stmt = SimpleStatement(
        "INSERT INTO notifications (user_id, notif_id, body, created_at) "
        "VALUES (%s, uuid(), %s, toTimestamp(now()))",
        consistency_level=ConsistencyLevel.ONE
    )
    try:
        session.execute(stmt, (user_id, body))
        return {"ok": True, "note": "written to local replica, will sync later"}
    except WriteTimeout:
        return {"ok": False, "reason": "even local replica unreachable"}

--- During London ↔ Frankfurt partition ---

safe_payment_write('acc_441', 4850.00):
  {'ok': False, 'reason': 'quorum unavailable — partition detected',
   'action': 'retry_after_partition_heals'}
  Behaviour: CP — refuses write, preserves consistency  ✓

safe_feed_write('usr_882', 'Alice liked your post'):
  {'ok': True, 'note': 'written to local replica, will sync later'}
  Behaviour: AP — accepts write, syncs when partition heals  ✓

After partition heals:
  Payment write retried successfully — balance correct
  Feed notification synced to Frankfurt replicas

Practice Questions — You're the Engineer

Quiz — Consistency vs Availability in Production