Key-Value Databases

Twitter's timeline service was once the slowest part of their stack. Loading a user's feed required assembling tweets from hundreds of accounts, sorting them, and returning thousands of rows — all under 100ms. It couldn't be done from a relational database at that scale. Their solution was a key-value layer in front of everything: pre-compute each user's timeline and store it behind a single key. Feed load time dropped from 800ms to 6ms. That's the key-value database doing exactly what it was built for.

How a Key-Value Store Works Under the Hood

At its core, a key-value store is a hash map — the same data structure you've used in every programming language. You give it a key, it hashes that key to a memory address, and reads or writes at that address directly. No scanning. No searching. No index traversal.

Redis — The Most Widely Used Key-Value Database

Redis (Remote Dictionary Server) is not just a key-value store — it's a data structure server. It supports strings, lists, sets, sorted sets, hashes, streams, and more — all stored in RAM, all accessible in sub-millisecond time. Redis has been the most-loved database in the Stack Overflow Developer Survey for 7 consecutive years.

Strings — The Foundation

The scenario: You're building a distributed lock system. Two instances of your order processing service might try to process the same order simultaneously. You need to ensure only one instance claims an order at a time. Redis SETNX (Set if Not eXists) is the standard solution:

import redis
r = redis.Redis(host='localhost', port=6379)

def claim_order(order_id, worker_id, timeout_seconds=30):
    key = f"lock:order:{order_id}"

    # SET key value NX EX — only set if key doesn't exist
    # NX = only write if key does NOT exist (atomic check + set)
    # EX = auto-expire after timeout_seconds (prevents deadlocks)
    acquired = r.set(key, worker_id, nx=True, ex=timeout_seconds)

    return acquired is not None  # True = lock claimed, False = already taken

# Two workers race for the same order
worker_a = claim_order('ord_8821', 'worker-1')
worker_b = claim_order('ord_8821', 'worker-2')

print(f"Worker A got lock: {worker_a}")
print(f"Worker B got lock: {worker_b}")

Worker A got lock: True    ← claimed first
Worker B got lock: False   ← key already exists, NX rejected the write

-- lock:order:ord_8821 holds "worker-1"
-- Expires in 30 seconds automatically
-- Worker A processes the order, then deletes the key
-- Worker B can retry after lock is released

Hashes — Objects Without Serialisation

Redis Hashes let you store a whole object under one key, with individual fields accessible without deserialising the entire value. This is more efficient than storing a JSON string when you only need to update one field.

The scenario: You're caching user profile data. Profiles are read on every page load. You want to update just the last_seen field without re-serialising the whole object:

# Store a user profile as a Hash — each field accessible individually
r.hset('user:u_4421', mapping={
    'name':       'Priya Sharma',
    'email':      'priya@example.com',
    'plan':       'pro',
    'login_count': '142'
})

# Update just one field — no need to read and rewrite the whole object
r.hset('user:u_4421', 'login_count', 143)

# Read just one field
plan = r.hget('user:u_4421', 'plan')

# Read all fields at once
profile = r.hgetall('user:u_4421')

plan = b'pro'

profile = {
  b'name':        b'Priya Sharma',
  b'email':       b'priya@example.com',
  b'plan':        b'pro',
  b'login_count': b'143'
}

Lists — Message Queues and Activity Feeds

The scenario: You're building a notification system. Background workers process notifications asynchronously. Producers push to the queue, consumers pop from it. Redis Lists give you a built-in queue with blocking pop — consumers wait efficiently without polling:

# Producer — push notification jobs to the right end of the list
def queue_notification(user_id, message):
    job = json.dumps({'user_id': user_id, 'msg': message})
    r.rpush('queue:notifications', job)  # RPUSH = push to Right end

# Push 3 jobs
queue_notification('u_001', 'Your order shipped!')
queue_notification('u_002', 'New follower: Priya')
queue_notification('u_003', '50% off — today only')

# Consumer — blocking pop from the left end
# BLPOP blocks for up to 30 seconds waiting for a job
# Returns immediately if a job is already in the queue
def process_notifications():
    while True:
        result = r.blpop('queue:notifications', timeout=30)
        if result:
            _, job_json = result
            job = json.loads(job_json)
            send_notification(job['user_id'], job['msg'])

-- Queue state after 3 pushes:
queue:notifications = [
  '{"user_id": "u_001", "msg": "Your order shipped!"}',   ← left (oldest)
  '{"user_id": "u_002", "msg": "New follower: Priya"}',
  '{"user_id": "u_003", "msg": "50% off — today only"}'   ← right (newest)
]

-- Consumer pops from left (FIFO order):
Processing: u_001 — "Your order shipped!"
Processing: u_002 — "New follower: Priya"
Processing: u_003 — "50% off — today only"

Persistence — RAM is Volatile, Redis Has a Plan

The obvious concern with an in-memory database: what happens when the server restarts? Redis has two persistence strategies — and most production deployments use both:

Key Expiry — The TTL System

TTL (Time To Live) is one of Redis's most powerful features. Every key can have an expiry time — Redis automatically deletes it when the time is up. No cron jobs. No cleanup scripts. Here's the full TTL toolkit:

# Set TTL at creation time
r.set('otp:u_4421', '847291', ex=300)        # expire in 5 minutes

# Add TTL to an existing key
r.expire('session:u_4421', 1800)             # expire in 30 minutes

# Check remaining TTL
ttl = r.ttl('otp:u_4421')                   # returns seconds remaining

# Make a key permanent again (remove TTL)
r.persist('session:u_4421')                 # key now lives forever

# Set expiry as an absolute Unix timestamp
r.expireat('report:jan_2024', 1735689600)   # expires Jan 1 2025

ttl = 287   # 287 seconds remaining (300 - 13 seconds elapsed)

# Special TTL return values:
# -1 = key exists with no expiry (persistent)
# -2 = key does not exist

Key Design Patterns — Naming Your Keys Well

Redis has no namespacing at the database level beyond 16 logical databases (db 0–15). Good key naming is how you organise everything in one Redis instance. The community standard is colon-separated namespacing:

Practice Questions — You're the Engineer

Quiz — Key-Value in Production