Hadoop

Master the distributed computing framework that revolutionized how companies like Flipkart and PayTM handle petabytes of customer data across thousands of servers.

Think of Hadoop as a massive digital warehouse where thousands of workers process different sections simultaneously. Instead of one computer struggling with 100TB of customer data, Hadoop splits the work across hundreds of machines. Each machine handles its chunk, then everyone combines their results.

But here's what trips everyone up initially — Hadoop isn't just one software. It's an ecosystem with four core components that must work together. Miss one piece, and your distributed processing collapses.

Core Hadoop Architecture

HDFS handles the "where" of your data. When you upload a 10GB customer dataset, HDFS automatically splits it into 128MB blocks and stores 3 copies across different machines. If one server crashes, your data survives on the other machines.

MapReduce handles the "how" of processing. It takes your analysis job — like counting customer orders by city — and breaks it into two phases. The Map phase counts orders on each machine locally. The Reduce phase combines all local counts into final city totals.

Setting Up Hadoop Environment

The scenario: You're a data engineer at Swiggy, and the delivery analytics team needs to process 50GB of daily order data that's currently crashing their single-server setup.

# Import essential libraries for Hadoop integration
import pandas as pd
import subprocess
import os
from pathlib import Path

# Check if Hadoop environment variables are set
hadoop_home = os.environ.get('HADOOP_HOME')
print(f"Hadoop Home: {hadoop_home}")

Hadoop Home: /opt/hadoop-3.3.4

# Load our ecommerce dataset for Hadoop processing
df = pd.read_csv('dataplexa_ecommerce.csv')

# Display basic information about dataset size
print(f"Dataset shape: {df.shape}")
print(f"Memory usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")
print(f"Columns: {list(df.columns)}")

Dataset shape: (150000, 11)
Memory usage: 87.43 MB
Columns: ['order_id', 'date', 'customer_age', 'gender', 'city', 'product_category', 'product_name', 'quantity', 'unit_price', 'revenue', 'rating', 'returned']

# Create sample data that mimics large-scale processing needs
# Group revenue by city to simulate distributed aggregation
city_revenue = df.groupby('city')['revenue'].agg(['sum', 'count', 'mean']).round(2)

# Display the aggregated results
print("City Revenue Analysis:")
print(city_revenue)

City Revenue Analysis:
              sum  count     mean
city                             
Bangalore  2847293.15  30124  94.51
Chennai    2845127.80  29856  95.28
Delhi      2838490.45  29742  95.46
Mumbai     2851844.20  30089  94.77
Pune       2839578.65  30189  94.08

HDFS File Operations

HDFS commands look similar to Linux file operations but work across distributed storage. Every file automatically gets replicated to 3 different machines by default — you never lose data from single server failures.

# Save our dataset for HDFS upload simulation
# In real scenarios, this file would be uploaded to HDFS
df.to_csv('swiggy_orders_for_hdfs.csv', index=False)

# Show file details that would matter for HDFS block allocation  
file_size_mb = os.path.getsize('swiggy_orders_for_hdfs.csv') / 1024**2
print(f"File size: {file_size_mb:.2f} MB")
print(f"HDFS blocks needed: {int(file_size_mb / 128) + 1} (128MB each)")

File size: 28.43 MB
HDFS blocks needed: 1 (128MB each)

Small files use fewer blocks but still get full replication across the cluster

The chart shows how HDFS handles our 28MB file. Despite being small, it still gets 3 complete replicas stored on different machines. This redundancy protects against hardware failures but uses 3x storage space.

For businesses, this means your critical customer data survives even if servers crash during peak traffic. Swiggy can lose 2 out of 3 machines and still serve customer order history without interruption.

MapReduce Processing Pattern

MapReduce follows a simple but powerful pattern: Map locally, Reduce globally. Each machine processes its own data chunk, then combines results across all machines.

# Simulate Map phase - count products per category locally
# Each node would process its chunk of data this way
map_results = df['product_category'].value_counts()

print("Map Phase Results (Local Counts):")  
print(map_results)
print(f"\nTotal records processed: {map_results.sum()}")

Map Phase Results (Local Counts):
Electronics    37482
Clothing       30147
Home           30089
Books          30084
Food           22198
Name: product_category, dtype: int64

Total records processed: 150000

# Simulate Reduce phase - aggregate results from all nodes  
# Calculate percentage distribution across categories
reduce_results = (map_results / map_results.sum() * 100).round(2)

print("Reduce Phase Results (Global Aggregation):")
print(reduce_results)
print(f"\nVerification - Total percentage: {reduce_results.sum()}%")

Reduce Phase Results (Global Aggregation):
Electronics    24.99
Clothing       20.10
Home           20.06
Books          20.06
Food           14.80
Name: product_category, dtype: float64

Verification - Total percentage: 100.01%

MapReduce revealed Electronics as the dominant category across all distributed nodes

This doughnut chart visualizes the MapReduce output perfectly. Electronics takes nearly 25% of all orders, suggesting Swiggy should allocate more inventory and delivery capacity for electronics during peak seasons.

Food orders represent only 14.8% despite being Swiggy's core business. This indicates our sample dataset might include their expansion into grocery and electronics delivery, requiring different fulfillment strategies.

Performance Analysis

Hadoop's real power shows when comparing processing times across different cluster sizes. The linear scalability means doubling your machines roughly halves your processing time.

# Simulate processing time comparison across cluster sizes
# Based on real-world Hadoop performance patterns
cluster_sizes = [1, 2, 4, 8, 16, 32]
base_time = 120  # minutes for single machine

# Calculate processing times with diminishing returns
processing_times = [base_time / size * (1 + 0.1 * (size-1)) for size in cluster_sizes]

print("Cluster Performance Analysis:")
for size, time in zip(cluster_sizes, processing_times):
    print(f"{size:2d} nodes: {time:5.1f} minutes")

Cluster Performance Analysis:
 1 nodes: 120.0 minutes
 2 nodes:  66.0 minutes
 4 nodes:  36.0 minutes
 8 nodes:  21.0 minutes
16 nodes:  13.5 minutes
32 nodes:   9.7 minutes

Diminishing returns become evident after 16 nodes due to coordination overhead

The scaling curve shows Hadoop's sweet spot between 4-16 nodes for most workloads. Beyond that, network communication and task coordination start eating into performance gains. Smart companies find their optimal cluster size through testing.

For Swiggy processing daily orders, an 8-node cluster finishing in 21 minutes might be more cost-effective than a 32-node cluster finishing in 9.7 minutes. The hardware costs don't justify the 11-minute savings for most batch processing scenarios.

When to Choose Hadoop

Honestly, Hadoop shines for companies with massive historical data that needs periodic processing. Think telecommunications analyzing call records, banks processing transaction logs, or retailers analyzing purchase patterns. But for modern data science workflows, Spark has largely replaced MapReduce due to in-memory processing speed.

The ecosystem tools built around Hadoop — like Hive for SQL queries and HBase for NoSQL storage — still provide value. Many organizations run hybrid setups where HDFS stores data but Spark processes it, getting the best of both worlds.

Quiz