NoSQL
Denormalization
Every database textbook teaches you to normalize — eliminate redundancy, store each fact once, keep your schema clean. Then you try to scale to 100 million users and discover that normalization is a read-time tax you can no longer afford. Denormalization is not laziness or bad practice. At scale, it is the deliberate engineering decision that keeps reads fast, servers cheap, and users happy.
What Normalization Costs at Scale
In a fully normalized relational schema, data is stored exactly once. To assemble a meaningful response, the database executes JOINs at query time — combining rows from multiple tables on every single request. At small scale, JOINs are essentially free. At large scale, they become the bottleneck.
The cost of a JOIN at scale — a concrete example:
❌ Normalised — JOINs at query time
SELECT p.body, u.name, u.avatar
FROM posts p
JOIN users u ON p.user_id = u.id
WHERE p.feed_id = ?
ORDER BY p.created_at DESC
LIMIT 20;
-- At 10B rows: index lookup on posts,
-- then 20 random seeks into users table.
-- Query time: 180ms under load
✅ Denormalised — data pre-joined at write time
{"{"}
body: "Just shipped v2.0...",
author_name: "Sarah Chen",
author_avatar: "cdn.../sarah.jpg",
created_at: ISODate("2025-01-15")
{"}"}
-- No JOIN needed. One collection read.
-- Query time: 3ms under load
The trade-off: if Sarah updates her display name, the normalised schema updates one row. The denormalised schema must update every post document Sarah ever wrote — potentially millions. You are trading write complexity for read speed. At read-heavy scale (most consumer apps read 10–100× more than they write), that trade is almost always worth it.
The Four Denormalization Techniques
Denormalization is not one thing — it is a family of techniques. Each solves a different performance problem.
Duplicate fields for read performance
Copy fields from a referenced document into the parent to avoid a second query. The referenced document remains the source of truth. The copy is a read-optimised snapshot. Used when the copied fields rarely change and are needed on every parent read.
Post stores author_name + author_avatar copied from the User document.
Pre-computed aggregates
Store COUNT, SUM, or AVG results directly in the document instead of computing them at query time. A product document stores review_count: 4821 rather than counting the reviews collection on every product page load. The aggregate is incremented on every write.
Product stores review_count and avg_rating instead of COUNT(reviews).
Duplicate tables for different access patterns
In Cassandra and DynamoDB, create multiple tables holding the same data structured around different query shapes. A trip exists in trips_by_driver and trips_by_city_date. Both are written on every trip completion. Each serves one access pattern perfectly.
Same trip data duplicated in three Cassandra tables for three query shapes.
Materialised views
Pre-compute the result of an expensive aggregation query and store it as a document that is refreshed on a schedule or on write. The read path hits the materialised view — a single document read — instead of running the aggregation pipeline every time.
Daily sales summary stored as one document, refreshed every hour by a background job.
Hands-on — Denormalizing a Product Catalogue
The scenario: You are the backend engineer for a marketplace with 5 million products and 200 million reviews. The product detail page is the most visited page on the site — 80 million page views per day. Every page load was running a COUNT and AVG aggregation against the reviews collection to display the star rating and review count. At peak, the reviews collection aggregation is taking 400ms and hammering the database. You are implementing pre-computed aggregates to fix it.
// BEFORE: product document with no denormalization
// Every page load runs: db.reviews.aggregate([{$match: {product_id}},
// {$group: {_id: null, avg: {$avg: "$rating"}, count: {$sum: 1}}}])
// At 200M reviews — 400ms per page load
db.products.insertOne({
_id: "prod_441",
name: "Mechanical Keyboard TKL",
price: 79.99,
category: "Electronics"
// No review stats — computed at query time (SLOW)
});{ acknowledged: true, insertedId: 'prod_441' }
// Page load time: 400ms (aggregation on 200M reviews) ❌The aggregation tax — why it kills page load time
Running COUNT and AVG on 200 million reviews on every product page load means scanning (at minimum) all reviews for that product on every request. Even with an index on product_id, returning from an aggregation pipeline is orders of magnitude slower than reading a single field from a single document.
The scenario continues: You redesign the product document to store pre-computed aggregates. The review stats are written to the product document every time a review is submitted — one extra write per review in exchange for eliminating the aggregation on every page load.
// AFTER: denormalized product document with pre-computed review stats
db.products.updateOne(
{ _id: "prod_441" },
{ $set: {
// Pre-computed aggregates — updated on every new review
"review_stats.count": 4821,
"review_stats.avg_rating": 4.3,
"review_stats.dist": { // rating distribution for histogram
"5": 2841,
"4": 1102,
"3": 512,
"2": 244,
"1": 122
}
}
}
);{ acknowledged: true, matchedCount: 1, modifiedCount: 1 }
// Page load time: 2ms (single document read) ✓"review_stats.count": 4821
The product document now carries its own aggregate. The page load reads one product document and gets the star rating, count, and histogram in the same response. Zero aggregation. Zero second query. 200× faster. The cost: every time someone submits a review, the write path must also update these three fields on the product.
"review_stats.dist": {"{"} "5": 2841, "4": 1102, ... {"}"}
The rating distribution histogram is also pre-computed — normally this would be a GROUP BY rating aggregation. Storing the distribution as a nested object means the product page can render the star histogram instantly. This is denormalization of an aggregation result — pre-computing what would otherwise be computed on every read.
The scenario continues: You now write the review submission handler. Every new review must update the reviews collection AND atomically update the pre-computed stats on the product document. You use MongoDB's $inc and $set operators to do both in one write path — no aggregation query needed to recalculate the average.
from pymongo import MongoClient
db = MongoClient()["marketplace"]
def submit_review(product_id, user_id, rating, body):
# Step 1: insert the review document
db.reviews.insert_one({
"product_id": product_id,
"user_id": user_id,
"rating": rating,
"body": body,
"created_at": datetime.utcnow()
})
# Step 2: update pre-computed stats on the product atomically
# Running count and sum allow us to recompute avg without a full scan
db.products.update_one(
{ "_id": product_id },
{
"$inc": {
"review_stats.count": 1,
"review_stats.rating_sum": rating, # track sum for avg
f"review_stats.dist.{rating}": 1 # histogram bucket
}
}
)
# Step 3: recompute avg from stored sum + count (no aggregation needed)
product = db.products.find_one(
{ "_id": product_id },
{ "review_stats.count": 1, "review_stats.rating_sum": 1 }
)
stats = product["review_stats"]
new_avg = round(stats["rating_sum"] / stats["count"], 1)
db.products.update_one(
{ "_id": product_id },
{ "$set": { "review_stats.avg_rating": new_avg } }
)Review submitted for prod_441 by user_7291 — rating: 5 reviews.insert_one: 1.1ms products.update_one ($inc): 0.9ms products.find_one (stats): 0.7ms products.update_one (avg): 0.8ms Total write path: 3.5ms — product page load remains: 2ms ✓
"review_stats.rating_sum" — the running sum trick
Storing the sum of all ratings alongside the count means you can recompute the average at any time as sum / count — with no aggregation query against the reviews collection. This is the key insight of pre-computed aggregates: never store just the derived value, store enough raw data to update it incrementally without a full scan.
$inc for atomic increments
MongoDB's $inc operator increments a field atomically at the database level — no read-modify-write cycle in application code. Under high concurrency (hundreds of reviews submitted per second), using $inc instead of a fetch-then-update pattern prevents lost updates where two simultaneous review submissions overwrite each other's count increment.
Hands-on — Denormalizing a Social Feed
The scenario: You are a senior engineer at a social platform. The user feed endpoint must return the 20 most recent posts from accounts a user follows, including each post's author name, avatar, and like count. The normalised approach requires: a JOIN on followers, a JOIN on users, and a COUNT on likes — three operations on some of the largest tables in the system. At 50 million active users each loading their feed 10 times a day, this is 500 million feed loads per day. You are redesigning the feed using a denormalised fan-out-on-write model.
def publish_post(author_id, body, db):
# Step 1: write the canonical post document
post = {
"_id": str(uuid.uuid4()),
"author_id": author_id,
"body": body,
"created_at": datetime.utcnow(),
"like_count": 0
}
db.posts.insert_one(post)
# Step 2: fetch the author's followers
author = db.users.find_one({"_id": author_id})
followers = db.followers.find({"following_id": author_id},
{"follower_id": 1})
# Step 3: fan-out — write a denormalised feed entry to every follower
# Each feed entry is a self-contained read — no JOIN needed at read time
feed_entries = [{
"user_id": f["follower_id"], # whose feed this is
"post_id": post["_id"],
"author_name": author["display_name"], # denormalised snapshot
"author_avatar": author["avatar_url"], # denormalised snapshot
"body": body, # denormalised copy
"created_at": post["created_at"],
"like_count": 0
} for f in followers]
if feed_entries:
db.user_feeds.insert_many(feed_entries)Post published by author_id: usr_882
Canonical post written: 1.2ms
Followers fetched: 2,400
Feed entries fan-out (bulk insert): 18.4ms
Total write path: ~20ms
Feed read for any follower:
db.user_feeds.find({user_id}).sort({created_at:-1}).limit(20)
Query time: 1.8ms (single collection, indexed) ✓Fan-out on write — the read-optimised feed model
When someone with 2,400 followers publishes a post, you write 2,400 feed entry documents — one per follower. The write is heavier (20ms instead of 1ms). But each of those 2,400 followers now loads their feed in 1.8ms with zero JOINs. Twitter, Instagram, and Facebook all use variants of this pattern. The write cost is paid once; the read savings are multiplied across every feed load.
The celebrity problem
A user with 50 million followers publishing one post would fan-out to 50 million documents — a write that takes minutes and floods the database. The real-world solution (used by Twitter) is a hybrid: fan-out on write for regular users, fan-out on read for celebrities above a follower threshold. This lesson covers the core pattern; hybrid approaches are a topic for Lesson 31 (Consistency vs Availability).
The Denormalization Trade-off Map
| Technique | Read benefit | Write cost | Consistency risk | When to use |
|---|---|---|---|---|
| Duplicate fields | Eliminates one JOIN / second query | Must update copies when source changes | Copies may lag source briefly | Fields rarely change (name, avatar) |
| Pre-computed aggregates | Eliminates COUNT/AVG aggregation | $inc on every write that affects the aggregate | Count/avg lags by one event under race condition | Aggregates displayed on high-traffic pages |
| Duplicate tables | Each table serves one query perfectly | N writes per event (one per table) | Tables can diverge if a write partially fails | Cassandra / DynamoDB multi-access-pattern |
| Materialised views | Complex aggregation → single doc read | Background job to refresh periodically | Data is stale by refresh interval | Dashboards, reports, tolerable staleness |
| Fan-out on write | Feed reads require zero JOINs | Write multiplied by follower count | Celebrity accounts cause write storms | Social feeds, activity streams |
Managing Consistency — Keeping Denormalised Data in Sync
The hardest part of denormalization is keeping copies consistent when the source data changes. Three patterns handle this in production:
Synchronous update in the same write path
When the source document changes, update all copies in the same application request. Simple, strongly consistent. Works when the fan-out is small (under a few hundred copies). Gets slow when there are thousands of copies to update.
Asynchronous via event queue
The write updates the source document and publishes a change event to a queue (Kafka, SQS). Background workers consume the event and update all copies. Eventually consistent — copies lag by seconds. Scales to millions of copies. Used by almost every large platform.
Accept controlled staleness
For some fields — like a display name or an avatar — brief staleness is acceptable. Old posts showing "Sarah Chen" for a few seconds after a name change is not a business problem. Define a staleness SLA and document it. Not everything needs to be in sync immediately.
Teacher's Note
Denormalization without a clear consistency plan is how you end up with a product that shows a user's bio from 2019 on posts they wrote last week. Before you denormalize any field, write down: what triggers an update to the copy, how the update is propagated, and what the acceptable staleness window is. If you cannot answer all three, you are not ready to denormalize that field yet.
Practice Questions — You're the Engineer
Scenario:
Scenario:
review_stats.count as a pre-computed aggregate. Under high traffic, hundreds of reviews are being submitted simultaneously. A junior developer wrote the update logic as: fetch the product, add 1 to the count in Python, then write the new value back. Under load you see the count drifting — it is lower than the actual number of reviews. You explain there is a single MongoDB operator that performs the increment atomically at the database level. What is it?
Scenario:
review_stats.avg_rating: 4.3 and review_stats.count: 4821. A new review comes in with a rating of 5. To update the average without running an aggregation over all 4,821 reviews, you need one additional field you stored alongside count. You use this field divided by count to recompute the average instantly. What is this field called in the lesson's example?
Quiz — Denormalization in Production
Scenario:
updateMany touching 2 million documents — taking 40 seconds and blocking the API response. What is the correct production pattern for propagating this update?
Scenario:
Scenario:
Up Next · Lesson 27
Partitioning & Sharding
How NoSQL databases split data across nodes — and why the key you choose determines whether your cluster scales linearly or collapses under load.