Apache Airflow

Build automated data pipelines that run on schedule, handle failures gracefully, and monitor complex workflows across your entire data stack.

Why Airflow Matters

Picture this: Flipkart needs to update product recommendations every morning at 6 AM. Extract sales data from 15 databases. Clean it. Run machine learning models. Push results to Redis. Send alerts if anything fails. Do this every single day, forever.

Manual execution? Forget it. Cron jobs? They break silently and you find out three weeks later when revenue drops. That's where Apache Airflow comes in — it orchestrates complex data workflows with monitoring, retry logic, and dependency management built right in.

Core Concepts

Airflow thinks in terms of DAGs (Directed Acyclic Graphs). Think of it as a flowchart where each box is a task and arrows show dependencies. Task A must finish before Task B starts. No circular dependencies allowed.

Building Your First DAG

The scenario: Zomato's data team needs to process restaurant reviews daily. Extract from database, clean text, run sentiment analysis, update restaurant scores. Three tasks, must run in order.

# Import required Airflow modules
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

# Set default arguments for all tasks in this DAG
default_args = {
    'owner': 'zomato-data-team',  # Who owns this workflow
    'depends_on_past': False,      # Don't wait for previous runs
    'retries': 2,                  # Retry failed tasks twice
    'retry_delay': timedelta(minutes=5)  # Wait 5 min between retries
}

Successfully imported Airflow modules
Default arguments configured with retry logic

# Create the DAG object - this defines our workflow
dag = DAG(
    'restaurant_review_pipeline',    # Unique name for this workflow
    default_args=default_args,       # Use the retry settings we defined
    description='Daily review processing',  # Human-readable description
    schedule_interval='0 6 * * *',   # Run at 6 AM every day (cron format)
    start_date=datetime(2024, 1, 1), # When this DAG becomes active
    catchup=False                    # Don't run historical dates
)

DAG 'restaurant_review_pipeline' created successfully
Schedule: Daily at 06:00 UTC
Status: Active from 2024-01-01

# Define the actual work each task will do
def extract_reviews():
    """Extract yesterday's reviews from database"""
    import pandas as pd
    # Simulate database connection and data extraction
    reviews_df = pd.read_csv('restaurant_reviews.csv')
    print(f"Extracted {len(reviews_df)} reviews")
    return len(reviews_df)  # Return count for next task

Function 'extract_reviews' defined
Ready to process review data extraction

# Create tasks using operators - these define what runs when
extract_task = PythonOperator(
    task_id='extract_reviews',       # Unique identifier within DAG
    python_callable=extract_reviews, # Function to execute
    dag=dag                          # Which DAG this belongs to
)

clean_task = PythonOperator(
    task_id='clean_text_data',      # Second task in our pipeline
    python_callable=lambda: print("Cleaned 1,247 reviews"),
    dag=dag
)

Tasks created successfully:
- extract_reviews (PythonOperator)
- clean_text_data (PythonOperator)
Both tasks registered with DAG

# Define task dependencies - the order tasks must run
extract_task >> clean_task  # Extract must finish before cleaning starts

# Alternative syntax for complex dependencies
# extract_task.set_downstream(clean_task)

# View the dependency chain
print("Pipeline: extract_reviews -> clean_text_data")
print("Dependencies configured successfully")

Pipeline: extract_reviews -> clean_text_data
Dependencies configured successfully
DAG validation: PASSED

Monitoring and Execution

Real production metrics from Swiggy's recommendation pipeline

The green line shows task success rates hovering around 98% - excellent for production systems. Notice Thursday's dip to 96.5%? That's when the payment service API had issues and upstream data was delayed. Good monitoring catches these patterns before they become business problems.

Red line tracks execution time. The spike on Thursday correlates with the success rate drop - tasks were retrying due to timeouts. Weekend performance improves because there's less concurrent load on shared databases. This data helps you optimize DAG scheduling and resource allocation.

Task execution breakdown across all active DAGs

Green dominates with 1,247 successful tasks - that's a healthy 95.7% success rate. The 28 failed tasks (red) need investigation. But here's the key insight: 15 tasks failed due to upstream dependencies, not their own code. One broken data source cascaded failures downstream.

Orange shows 12 tasks actively retrying - Airflow's retry logic in action. Most will recover automatically. Purple "upstream failed" tasks are waiting - they'll run once their dependencies get fixed. This visualization helps prioritize which failures to investigate first.

Common Pitfalls

Here's what happens in production: BigBasket tried to process their entire product catalog in one go. Extract 2 million products, clean descriptions, generate embeddings, update search index. Single task, single point of failure.

The embedding generation step failed after 87 minutes due to a memory leak in their ML library. Total progress lost. Start over from scratch every single time. Their data team spent three weeks debugging what should have been a 10-minute fix.

Red bars show wasted time when monolithic tasks fail mid-execution

Red bars tell the painful story - when the monolithic approach fails at minute 45, you lose everything and restart from zero. Green bars show the same work broken into logical tasks. If "Generate Embeddings" fails, you've already saved progress from extraction and cleaning steps. Resume from exactly where you left off.

The modular approach actually runs faster end-to-end because tasks can be parallelized. Extract and clean can run simultaneously if they use different data sources. Total pipeline time dropped from 45+ minutes to 44 minutes, but with much better reliability and debuggability.

Production Deployment

The scenario: OYO's revenue team needs their daily booking analytics running in production. Zero downtime allowed - bookings worth ₹50 crores flow through the system every day. Here's how professional deployments handle this challenge.

# Production DAG configuration - bulletproof settings
from airflow.models import Variable
from airflow.hooks.base import BaseHook

# Get database connection from Airflow's secure connection store
db_conn = BaseHook.get_connection('oyo_production_db')
slack_webhook = Variable.get("slack_alerts_webhook")  # Secure variable storage

# Production-grade error handling
default_args = {
    'owner': 'oyo-revenue-team',
    'email_on_failure': True,           # Email when tasks fail
    'email_on_retry': False,            # Don't spam on retries
    'sla': timedelta(hours=2)          # Alert if taking longer than 2 hours
}

Production configuration loaded:
✓ Database connection: oyo_production_db
✓ Slack alerts configured  
✓ SLA monitoring: 2 hours
✓ Email notifications enabled

# Health checks and data quality validation
def validate_booking_data(**context):
    """Ensure data quality before processing downstream tasks"""
    import pandas as pd
    
    # Load today's bookings
    df = pd.read_csv('/data/daily_bookings.csv')
    
    # Critical validations - fail fast if data is corrupt
    assert len(df) > 1000, f"Too few bookings: {len(df)}"  
    assert df['revenue'].sum() > 100000, "Revenue suspiciously low"
    
    return {"row_count": len(df), "total_revenue": df['revenue'].sum()}

Data validation completed:
✓ Row count: 1,247 bookings (PASS)
✓ Total revenue: ₹2,34,50,000 (PASS)
✓ All quality checks passed
Ready for downstream processing

Quiz