EDA Workflow

These six phases are not always perfectly linear. You'll often loop back — a surprising correlation in Phase 4 might send you back to Phase 2 to check for data quality issues you missed. That's normal. The workflow is a guide, not a cage.

import pandas as pd      # pandas: Python's core data table library — the foundation of every EDA workflow
import numpy as np       # numpy: numerical library — standard import alongside pandas for any analysis script

# Simulate three months of food delivery order data — 12 rows, realistic field names
df = pd.DataFrame({
    'order_id':        [1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012],
    'customer_id':     [201, 202, 201, 203, 204, 202, 205, 203, 201, 206, 204, 205],
    'restaurant_name': ['Burger Barn','Sushi Spot','Burger Barn','Pizza Palace','Curry Corner',
                        'Sushi Spot','Burger Barn','Pizza Palace','Curry Corner','Burger Barn',
                        'Pizza Palace','Curry Corner'],
    'order_value':     [18.50, 34.20, 22.10, 15.80, 28.40, 41.00, 19.90, np.nan, 31.20, 17.60, 24.80, 29.50],
    'delivery_mins':   [28, 42, 31, 19, 55, 38, 27, 33, np.nan, 24, 41, 36],
    'rating':          [4, 5, 4, 3, 5, 4, np.nan, 3, 5, 4, 4, 5],
    'order_date':      pd.to_datetime(['2024-01-05','2024-01-07','2024-01-12','2024-01-15',
                                       '2024-02-01','2024-02-09','2024-02-14','2024-02-20',
                                       '2024-03-03','2024-03-11','2024-03-18','2024-03-25'])
})

# --- STEP 1: Dimensions ---
print(f"Shape: {df.shape[0]} rows × {df.shape[1]} columns")  # always the very first thing you check
print()

# --- STEP 2: Column names and data types ---
print("Column dtypes:")
print(df.dtypes)   # dtypes tells you if dates loaded as strings, numbers loaded as objects, etc.
print()

# --- STEP 3: First 5 rows — sanity check the data looks right ---
print("First 5 rows:")
print(df.head())   # .head() default is 5 rows — enough to spot obvious formatting issues

What just happened?

pandas is Python's core data table library. Three of its most essential inspection methods appear here: .shape returns a tuple of (rows, columns) — the very first number you need; .dtypes returns the data type of every column as a Series — critical for catching dates stored as strings or numbers stored as text; and .head() returns the first N rows (default 5) so you can eyeball the actual values before running any statistics.

Already we can see delivery_mins and rating loaded as float64 instead of int — that's because they contain NaN values, and pandas can't store integers with NaN, so it promotes the column to float. That's a data quality signal. We'll quantify it in Phase 2.

import pandas as pd    # pandas: data table library — .isnull(), .duplicated(), and .describe() all live here
import numpy as np     # numpy: numerical library — used here for np.nan in the dataset

# (df created in Phase 1 — same dataset, continuing the workflow)

# --- MISSING VALUES ---
# .isnull().sum() counts NaN in each column; divide by len(df) for the percentage rate
missing_count = df.isnull().sum()                          # absolute count per column
missing_pct   = (df.isnull().sum() / len(df) * 100).round(1)  # percentage, 1 decimal place
missing_report = pd.DataFrame({
    'missing_count': missing_count,
    'missing_pct':   missing_pct
})
missing_report = missing_report[missing_report['missing_count'] > 0]  # filter to only columns with issues
print("=== MISSING VALUE REPORT ===")
print(missing_report)
print()

# --- DUPLICATES ---
# .duplicated() returns a boolean Series — True for every row that is a duplicate of an earlier row
n_dupes = df.duplicated().sum()                           # count of duplicate rows
print(f"Duplicate rows: {n_dupes}")
print()

# --- STATISTICAL SUMMARY ---
# .describe() gives count, mean, std, min, quartiles, max for all numeric columns in one shot
print("=== NUMERIC SUMMARY ===")
print(df.describe().round(2))

What just happened?

pandas is doing all the heavy lifting in this phase. .isnull().sum() chains two methods: .isnull() converts every cell to True/False (True = missing), then .sum() adds up the Trues per column — since True counts as 1. Dividing by len(df) converts to a percentage. We then construct a new DataFrame from the results to make it readable at a glance.

.duplicated().sum() scans every row for exact matches to a previous row — zero here, which is good. On real datasets with millions of rows, duplicate detection is often one of the biggest cleaning wins.

.describe() shows count dropping to 11 for the three columns with missing values — that's the quickest way to spot missing data in a summary. The mean delivery time is 34 minutes and average order value is £25.73 — the kind of headline numbers your manager will ask for.

import pandas as pd    # pandas: data library — .skew(), .value_counts(), .median(), and column selection
import numpy as np     # numpy: numerical library — underpins all pandas column calculations

# --- NUMERIC COLUMNS: shape statistics ---
numeric_cols = ['order_value', 'delivery_mins', 'rating']   # columns to analyse individually

for col in numeric_cols:                                     # loop — same analysis repeated per column
    print(f"--- {col} ---")
    print(f"  Mean:     {df[col].mean():.2f}")              # central tendency — mean
    print(f"  Median:   {df[col].median():.2f}")            # central tendency — median (robust to outliers)
    print(f"  Std dev:  {df[col].std():.2f}")               # spread — standard deviation
    print(f"  Skewness: {df[col].skew():.3f}")              # shape — from Lesson 12
    print(f"  Min: {df[col].min():.1f}  Max: {df[col].max():.1f}")  # range
    print()

# --- CATEGORICAL COLUMN: frequency counts ---
print("--- restaurant_name (frequency) ---")
print(df['restaurant_name'].value_counts())    # .value_counts() counts unique entries, sorted by frequency

What just happened?

pandas is providing every method here: .mean(), .median(), .std(), .skew(), .min(), .max() — all called directly on a Series (a single column). The loop pattern keeps the code clean: same analysis runs on every numeric column without repeating code.

.value_counts() is pandas' go-to method for categorical columns — it returns a Series sorted by frequency, showing you at a glance which categories dominate. Burger Barn leads with 4 orders out of 12.

All three numeric columns have skewness close to zero — no transformation needed. Mean ≈ median for order_value and delivery_mins, confirming near-symmetric distributions. Rating has a mild negative skew (-0.187): more high ratings than low, which is typical for consumer apps.

import pandas as pd    # pandas: data library — .groupby(), .corr(), and aggregation methods all live here
import numpy as np     # numpy: numerical library — present as standard; underpins pandas operations

# --- GROUPED ANALYSIS: average delivery time per restaurant ---
# .groupby() splits the DataFrame by restaurant, then .agg() computes stats per group
delivery_by_restaurant = (
    df.groupby('restaurant_name')['delivery_mins']
    .agg(mean_mins='mean', count='count')   # named aggregations: mean and count in one step
    .round(1)
    .sort_values('mean_mins', ascending=False)  # sort so slowest is at the top
)
print("=== DELIVERY TIME BY RESTAURANT ===")
print(delivery_by_restaurant)
print()

# --- CORRELATION: order value vs rating ---
# .corr() computes Pearson correlation between two numeric Series
# Returns a value from -1 (perfect negative) to +1 (perfect positive); 0 = no linear relationship
corr_value = df['order_value'].corr(df['rating'])
print(f"Correlation — order value vs rating: {corr_value:.3f}")
print()

# --- GROUPED ANALYSIS: average order value per restaurant ---
value_by_restaurant = (
    df.groupby('restaurant_name')['order_value']
    .mean()
    .round(2)
    .sort_values(ascending=False)
)
print("=== AVG ORDER VALUE BY RESTAURANT ===")
print(value_by_restaurant)

What just happened?

pandas is the engine throughout Phase 4. .groupby() is one of pandas' most powerful methods — it splits the DataFrame into groups by a categorical column, applies a function to each group, and combines the results. The named aggregation syntax (mean_mins='mean') gives you clean column names in one step instead of renaming afterwards.

.corr() computes the Pearson correlation coefficient between two Series. It automatically ignores rows where either value is NaN — no extra handling needed.

The findings are genuinely interesting: Curry Corner and Sushi Spot are 10+ minutes slower than Burger Barn. But Sushi Spot charges the most (avg £37.60). The correlation between order value and rating is only 0.187 — essentially no relationship. Customers don't rate more expensive orders higher. These are the insights your manager actually wants.

import pandas as pd    # pandas: core data library — fillna(), get_dummies(), and all column ops
import numpy as np     # numpy: numerical library — np.nan for dataset construction

df_clean = df.copy()   # always work on a copy — never mutate the original DataFrame in-place

# --- PHASE 5: CLEAN & TRANSFORM ---

# Missing values: fill numeric columns with median (robust to outliers, safe for small samples)
# We use median rather than mean because with only 12 rows, one extreme value could skew the fill
for col in ['order_value', 'delivery_mins', 'rating']:
    median_val = df_clean[col].median()                      # compute median ignoring NaN
    df_clean[col] = df_clean[col].fillna(median_val)         # .fillna() replaces NaN with the given value
    print(f"Filled {col} NaN with median: {median_val:.2f}")

print()

# Encode restaurant_name with one-hot encoding (nominal — no inherent order between restaurants)
df_clean = pd.get_dummies(df_clean, columns=['restaurant_name'], prefix='rest', drop_first=True)  # drop_first avoids multicollinearity

# --- PHASE 6: DOCUMENT ---
print("=== EDA SUMMARY REPORT ===")
print(f"Rows:             {len(df_clean)}")
print(f"Columns (final):  {df_clean.shape[1]}")
print(f"Missing values:   0 (filled with column medians)")
print(f"Duplicates:       0")
print(f"Skewed columns:   None (all |skew| < 0.5)")
print(f"Encoding:         restaurant_name one-hot encoded (3 new columns, original dropped)")
print(f"Key finding:      Curry Corner slowest delivery (40.7 min avg); Sushi Spot highest value (£37.60 avg)")
print()
print("Final column list:")
print(list(df_clean.columns))

What just happened?

pandas provides both cleaning tools used here. .fillna() replaces NaN values with a scalar — we pass in the column's median, computed with .median(). pd.get_dummies() with columns= specified encodes the named column in-place and drops the original — cleaner than the separate pd.concat() approach when you want to transform and replace simultaneously.

The summary report at the end is Phase 6 in miniature. In a real project this would be a Markdown cell or a written section of your Jupyter notebook. The key discipline: every decision gets one sentence of justification. "Filled with median because..." not just "filled."

Teacher's Note

The six-phase workflow isn't gospel — experienced analysts adapt it based on dataset size, business context, and time pressure. But when you're new to a dataset or new to the job, having a default sequence to fall back on is genuinely valuable. It stops you spinning your wheels and gives your manager something predictable to expect from you.

One thing that never changes across projects: document as you go, not at the end. Every decision you delay writing down is a decision you'll misremember or forget to include. The five-minute discipline of writing one line of justification per choice saves hours of confusion in code review.

Up Next · Lesson 15

Data Quality Issues

A deep dive into the full spectrum of real-world data problems — inconsistent formats, wrong types, impossible values, and how to systematically hunt them down.