Backup & Recovery

Every engineer knows backups are important. Far fewer have actually restored from one under pressure, at 2am, with a VP on the phone asking when the data will be back. A backup strategy that has never been tested is not a strategy — it is a hope. This lesson is about building something you can actually rely on when the worst happens.

Why NoSQL Backup Is Different

Relational databases have decades of mature backup tooling. NoSQL systems are more varied — a Cassandra cluster, a MongoDB replica set, and a DynamoDB table each require completely different backup approaches. Three concerns are common to all of them:

Method 1 — mongodump and mongorestore

mongodump is MongoDB's native logical backup tool. It connects to a running MongoDB instance and exports collections as BSON files. It is simple, portable, and works without any cloud infrastructure. The trade-off: for large databases it is slow, and restoring a large dump to a fresh instance takes significant time.

The scenario: You are the sole backend engineer at an early-stage startup. Your MongoDB database holds 3 months of user data — about 8GB. You have no cloud snapshot setup yet. You need a nightly backup script that runs from a secondary replica (to avoid hitting the primary), compresses the output, uploads it to S3, and retains the last 7 days. You need this running before you go to sleep tonight.

#!/bin/bash
set -euo pipefail   # exit on error, unset vars, pipe failures

DATE=$(date +%Y-%m-%d)
BACKUP_DIR="/tmp/mongo-backup-$DATE"
S3_BUCKET="s3://myapp-backups/mongodb"
RETENTION_DAYS=7

echo "[$DATE] Starting mongodump from secondary..."

# --readPreference=secondary — dump from replica, not primary
# --gzip — compress BSON files inline (reduces size ~70%)
# --oplog — capture oplog entries for a point-in-time consistent backup
mongodump \
  --uri="mongodb://backup_user:$MONGO_PASS@mongo-secondary:27017" \
  --readPreference=secondary \
  --gzip \
  --oplog \
  --out="$BACKUP_DIR"

echo "Dump complete. Uploading to S3..."

# Tar the directory and stream directly to S3 (no extra disk space needed)
tar -czf - "$BACKUP_DIR" | aws s3 cp - "$S3_BUCKET/backup-$DATE.tar.gz"

# Remove local temp files
rm -rf "$BACKUP_DIR"

# Delete S3 objects older than retention window
aws s3 ls "$S3_BUCKET/" \
  | awk '{print $4}' \
  | while read key; do
      aws s3 rm "$S3_BUCKET/$key" \
        --expires-before "$(date -d "-$RETENTION_DAYS days" +%Y-%m-%d)" \
        2>/dev/null || true
    done

echo "[$DATE] Backup complete. Stored at $S3_BUCKET/backup-$DATE.tar.gz"

[2025-03-10] Starting mongodump from secondary...
2025-03-10T02:00:01.441+0000  writing myapp.users to dump/myapp/users.bson
2025-03-10T02:00:04.112+0000  done dumping myapp.users (184203 documents)
2025-03-10T02:00:04.113+0000  writing myapp.orders to dump/myapp/orders.bson
2025-03-10T02:00:09.887+0000  done dumping myapp.orders (442891 documents)
2025-03-10T02:00:09.901+0000  writing captured oplog  ✓
Dump complete. Uploading to S3...
upload: - to s3://myapp-backups/mongodb/backup-2025-03-10.tar.gz
[2025-03-10] Backup complete. Stored at s3://myapp-backups/mongodb/backup-2025-03-10.tar.gz
Elapsed: 4m 12s  |  Size: 2.1GB (compressed from 8.3GB)

Restoring from a mongodump Backup

Knowing how to take a backup matters. Knowing how to restore from one matters more. Here is the restore process for the backup we just created — including the oplog replay for point-in-time consistency.

The scenario: A developer accidentally ran a script that dropped the orders collection in production. The incident happened at 03:47 UTC. Your last backup was taken at 02:00 UTC and completed at 02:04 UTC. You need to restore the orders collection to its exact state at 02:04 UTC — the moment the backup completed — without touching any other collections.

# Step 1: download the backup from S3
aws s3 cp s3://myapp-backups/mongodb/backup-2025-03-10.tar.gz /tmp/
tar -xzf /tmp/backup-2025-03-10.tar.gz -C /tmp/

# Step 2: restore only the orders collection
# --nsInclude — restore only this namespace (db.collection)
# --oplogReplay — replay the captured oplog for consistency
# --drop — drop the existing (empty) collection before restore
mongorestore \
  --uri="mongodb://admin:$MONGO_PASS@mongo-primary:27017" \
  --nsInclude="myapp.orders" \
  --oplogReplay \
  --drop \
  --gzip \
  /tmp/mongo-backup-2025-03-10/

echo "Restore complete. Validating document count..."

# Step 3: sanity check — count should match pre-incident count
mongo myapp --eval "db.orders.countDocuments()"

2025-03-10T03:55:12+0000  restoring myapp.orders from orders.bson.gz
2025-03-10T03:55:18+0000  442891 document(s) restored successfully
2025-03-10T03:55:18+0000  replaying oplog — 1247 entries applied  ✓
Restore complete. Validating document count...
442891   ✓  matches pre-incident count

Total restore time: 6m 44s
Orders collection live again at 04:01 UTC
Incident window: 03:47 → 04:01 (14 minutes)

Method 2 — Cloud Snapshots (Atlas / EBS)

Logical backups like mongodump are flexible but slow. For large databases, volume snapshots are faster by an order of magnitude — because they operate at the storage block level rather than reading every document through the database engine. MongoDB Atlas continuous cloud backups and AWS EBS snapshots both use this approach.

The scenario: Your team has migrated to MongoDB Atlas. Your database is now 200GB and growing. A mongodump takes 3 hours and restoration takes 5 hours — far beyond your 1-hour RTO. You are switching to Atlas continuous cloud backups with point-in-time recovery and need to verify the configuration via the Atlas API.

PROJECT_ID="abc123"
CLUSTER="prod-cluster"
API_BASE="https://cloud.mongodb.com/api/atlas/v1.0"

# Enable continuous cloud backups with 7-day point-in-time window
curl -s -u "$ATLAS_PUBLIC:$ATLAS_PRIVATE" --digest \
  -X PATCH "$API_BASE/groups/$PROJECT_ID/clusters/$CLUSTER" \
  -H "Content-Type: application/json" \
  -d '{
    "providerBackupEnabled": true,
    "pitEnabled": true
  }'

# List the most recent snapshots to verify backups are running
curl -s -u "$ATLAS_PUBLIC:$ATLAS_PRIVATE" --digest \
  "$API_BASE/groups/$PROJECT_ID/clusters/$CLUSTER/backup/snapshots" \
  | jq '.results[] | {id, createdAt, storageSizeBytes, status}'

{
  "id": "snap_20250310_020000",
  "createdAt": "2025-03-10T02:00:00Z",
  "storageSizeBytes": 214748364800,   // 200GB
  "status": "completed"
}
{
  "id": "snap_20250309_020000",
  "createdAt": "2025-03-09T02:00:00Z",
  "storageSizeBytes": 209715200000,
  "status": "completed"
}

Continuous backup: ENABLED  ✓
Point-in-time recovery window: 7 days  ✓
Latest snapshot: 2025-03-10T02:00:00Z  ✓

Backup for Cassandra — nodetool and Snapshots

Cassandra's backup approach is fundamentally different from MongoDB's. Cassandra uses an LSM-tree storage engine — data is written to immutable SSTable files and never modified in place. This makes backups simpler in one way: you can copy SSTables directly because they do not change after they are written. The challenge is consistency across a multi-node cluster.

The scenario: You are a platform engineer managing a 6-node Cassandra cluster storing IoT sensor data. You need to take a consistent snapshot across all nodes simultaneously and back it up to object storage. A snapshot taken one node at a time would capture different points in time — useless for a consistent restore.

SNAPSHOT_TAG="backup-$(date +%Y%m%d)"
NODES=("cass-node-1" "cass-node-2" "cass-node-3" "cass-node-4" "cass-node-5" "cass-node-6")
KEYSPACE="iot_sensors"

echo "Triggering snapshot on all nodes simultaneously..."

# Run nodetool snapshot on all nodes in parallel
# -t = tag name for the snapshot
for node in "${NODES[@]}"; do
  ssh "$node" "nodetool snapshot -t $SNAPSHOT_TAG $KEYSPACE" &
done
wait   # wait for all parallel snapshots to complete

echo "All nodes snapshotted. Uploading SSTables to S3..."

for node in "${NODES[@]}"; do
  # Cassandra stores snapshots inside the data directory
  ssh "$node" \
    "find /var/lib/cassandra/data/$KEYSPACE -name 'snapshots/$SNAPSHOT_TAG' \
     -exec tar -czf - {} \; \
     | aws s3 cp - s3://iot-backups/cassandra/$node-$SNAPSHOT_TAG.tar.gz" &
done
wait

echo "Snapshot $SNAPSHOT_TAG complete across all nodes."

Triggering snapshot on all nodes simultaneously...
[cass-node-1] Requested creating snapshot(s) for [iot_sensors]
[cass-node-2] Requested creating snapshot(s) for [iot_sensors]
[cass-node-3] Requested creating snapshot(s) for [iot_sensors]
[cass-node-4] Requested creating snapshot(s) for [iot_sensors]
[cass-node-5] Requested creating snapshot(s) for [iot_sensors]
[cass-node-6] Requested creating snapshot(s) for [iot_sensors]
All nodes snapshotted. Uploading SSTables to S3...
Uploaded: cass-node-1-backup-20250310.tar.gz  (34.2GB)
Uploaded: cass-node-2-backup-20250310.tar.gz  (33.9GB)
... (all 6 nodes)
Snapshot backup-20250310 complete across all nodes.
Total backup size: 204GB  |  Elapsed: 8m 21s

Testing Your Recovery — The Most Important Step

A backup you have never restored from is not a backup — it is a hypothesis. The only way to know your backup works is to restore it, validate the data, and measure how long it took. This should be a scheduled, automated process — not something you discover is broken at 3am during an actual incident.

Practice Questions — You're the Engineer

Quiz — Backup & Recovery in Production