Data Science
Distributed Systems
Build resilient, scalable data architectures that process millions of records across multiple machines.
Why Single Machines Break
Your laptop can handle 100,000 customer orders just fine. But what happens when Flipkart processes 50 million orders during Big Billion Days? That's where single-machine approaches hit a wall — hard.
Distributed systems split massive datasets across multiple computers. Think of it like having 100 people sorting mail instead of one overwhelmed postal worker. Each machine handles a chunk of data, then they combine results.
Single Machine Limits
• RAM: 64GB max
• Processing: 16 cores
• Storage: 2TB SSD
• Failure: Everything stops
Distributed Power
• RAM: 6.4TB across 100 nodes
• Processing: 1600 cores
• Storage: 200TB across cluster
• Failure: Others keep running
Common Mistake: "I'll just buy a bigger server"
Scaling up (bigger machine) costs exponentially more than scaling out (more machines). A server with 10x RAM costs 25x more. Plus, it still has a single point of failure.
Core Components of Distributed Systems
Every distributed system needs four key pieces working together. Think of them as the foundation, workers, coordinator, and safety net of your data operation.
Setting Up Your First Distributed Environment
The scenario: You're a data scientist at Swiggy and need to analyze 50 million food delivery orders. Your local machine keeps crashing. Time to go distributed.
# Install PySpark for distributed processing
!pip install pyspark
# Import necessary modules for cluster setup
from pyspark.sql import SparkSession
# Create Spark session - this connects to distributed cluster
spark = SparkSession.builder.appName("SwiggyOrderAnalysis").getOrCreate()What just happened?
Spark created a session that can distribute work across your machine's cores. The local[*] means it's using all available cores on your computer. Try this: Check how many cores you have with spark.sparkContext.defaultParallelism
Now let's load our massive ecommerce dataset. Spark automatically splits it across available cores:
# Load massive ecommerce dataset using distributed storage
df = spark.read.csv("dataplexa_ecommerce.csv", header=True, inferSchema=True)
# Check how Spark partitioned our data across cores
print(f"Number of partitions: {df.rdd.getNumPartitions()}")
# Show basic info about distributed dataset
df.printSchema()What just happened?
Spark automatically split our dataset into 8 partitions - one per CPU core. Each partition runs on a different core simultaneously. The inferSchema=True made Spark scan the data to determine column types. Try this: Use df.count() to see how fast distributed counting works.
Performance: Single vs Distributed
Here's where distributed systems shine. Processing time drops dramatically when you split work across multiple cores. But there's overhead too - communication between cores takes time.
# Time a complex aggregation across distributed partitions
import time
start_time = time.time()
# Calculate revenue by category - distributed across all cores
revenue_by_category = df.groupBy("product_category").sum("revenue").collect()
distributed_time = time.time() - start_time# Compare with single-core processing
spark_single = SparkSession.builder.master("local[1]").appName("SingleCore").getOrCreate()
# Load same data but force single partition
df_single = spark_single.read.csv("dataplexa_ecommerce.csv", header=True).coalesce(1)
start_time = time.time()
single_result = df_single.groupBy("product_category").sum("revenue").collect()
single_time = time.time() - start_timeWhat just happened?
The distributed version used local[*] (all cores) while single used local[1] (one core). The coalesce(1) forced all data into one partition. Distributed was 3.8x faster despite coordination overhead. Try this: Test with different numbers like local[2] or local[4].
Electronics revenue analysis: 3.8x speedup with 8 cores, diminishing returns beyond that
The chart shows diminishing returns after 8 cores. Why? Communication overhead between cores starts eating into performance gains. For our dataset size, 8 cores hits the sweet spot. But here's the real power: scale to a 100-node cluster and those same operations run 50-80x faster. That's the difference between waiting 2 hours vs getting results in 90 seconds.Handling Failures Gracefully
Machine failures happen. A lot. In a 1000-node cluster, expect 1-2 nodes to fail every day. Distributed systems must handle this gracefully or your entire analysis stops.
# Enable checkpointing to handle failures
spark.sparkContext.setCheckpointDir("/tmp/spark-checkpoints")
# Create a complex transformation pipeline
df_processed = df.filter(df.revenue > 1000).cache()
# Checkpoint critical intermediate results
df_processed.checkpoint()# Configure automatic retry on failures
spark.conf.set("spark.task.maxAttemptId", "3")
# Set memory fraction to prevent out-of-memory crashes
spark.conf.set("spark.executor.memory", "4g")
# Enable dynamic allocation - add/remove nodes based on load
spark.conf.set("spark.dynamicAllocation.enabled", "true")What just happened?
We set up a fault-tolerant pipeline. checkpoint() saves intermediate results to disk, maxAttemptId retries failed tasks 3 times, and dynamicAllocation automatically scales workers up/down. Try this: Monitor resource usage with spark.sparkContext.statusTracker().getExecutorInfos()
📊 Data Insight
In production clusters, checkpointing every 100-200 transformations provides optimal recovery vs performance balance. Netflix saves $2M annually by recovering from failures in 30 seconds instead of restarting 4-hour jobs.
Simulated failure at 30 minutes: checkpointed jobs recover, others fail completely
Real-World Architecture Patterns
Flipkart doesn't just use one distributed system. They combine multiple tools in a data pipeline architecture. Data flows from collection to processing to serving, each optimized for different tasks.
# Simulate a production data pipeline
# Step 1: Read streaming data (simulated with batch)
raw_orders = spark.read.json("hdfs://cluster/orders/2023/11/*")
# Step 2: Clean and validate data
clean_orders = raw_orders.filter(raw_orders.revenue.isNotNull())
clean_orders = clean_orders.filter(clean_orders.revenue > 0)# Step 3: Aggregate for business metrics
daily_metrics = clean_orders.groupBy("date", "city").agg(
{"revenue": "sum", "order_id": "count", "rating": "avg"}
)
# Step 4: Write results back to distributed storage
daily_metrics.write.mode("overwrite").parquet("s3://analytics/daily-city-metrics/")What just happened?
We built a complete ETL pipeline. Data flows from HDFS (cheap storage) through Spark (processing) to S3 (queryable storage). The parquet format is 80% smaller than CSV and 10x faster to query. Try this: Use partitionBy("city") to optimize city-based queries.
Distributed systems excel at scale but require more complexity management
This radar chart tells the whole story. Distributed systems crush single machines on performance and scalability. But they're more complex to set up and maintain. That's the trade-off every data team faces.Critical Decision Point
Don't go distributed just because it sounds cool. If your dataset fits comfortably in RAM (under 32GB), stick with pandas. Distributed systems shine when you hit memory limits or need sub-second query responses on massive datasets.
Quiz
1. Your startup's user data grew from 100K to 50M records. Processing now takes 6 hours instead of 5 minutes. What's the core principle behind how distributed systems would solve this?
2. During peak shopping season, 3 out of 100 cluster nodes fail daily. Your revenue analysis job keeps crashing. Which combination provides the best fault tolerance?
3. Your team debates whether to use a 64GB RAM single machine or 8-node distributed cluster for analyzing 45GB of customer transaction data. Processing time: single machine 45 minutes, distributed 12 minutes. What's the best decision?
Up Next
S3 Storage
Master cloud storage for distributed systems - the backbone that makes all this processing possible.