Hypothesis Testing

Master statistical hypothesis testing to validate business assumptions and make data-driven decisions with confidence.

What Is Hypothesis Testing?

Think of hypothesis testing like a court trial for your data. You start with an assumption (the defendant is innocent), gather evidence, and decide whether that evidence is strong enough to reject your original assumption.

Every hypothesis test has two competing statements. The null hypothesis (H₀) represents the status quo - what you assume is true until proven otherwise. The alternative hypothesis (H₁) is what you're trying to prove.

Honestly, this trips up 80% of beginners. They think we're trying to "prove" the alternative hypothesis. Wrong. We're trying to find enough evidence to reject the null hypothesis. Big difference.

Common Statistical Tests

Choosing the right test depends on your data type and what you're comparing. Here's the breakdown that covers 90% of real-world scenarios:

Hands-On: One-Sample t-Test

The scenario: Flipkart's analytics team claims their average order value is ₹1,500. A data scientist needs to verify this claim using recent transaction data. The stakes? Marketing budget allocation for the next quarter.

# Import statistical testing library
import pandas as pd
import numpy as np
from scipy import stats

# Load the ecommerce dataset
df = pd.read_csv('dataplexa_ecommerce.csv')

# Display first few rows to understand structure
print(df.head())

   order_id        date  customer_age gender        city product_category
0      1001  2023-01-05            34      M      Mumbai      Electronics
1      1002  2023-01-05            28      F       Delhi         Clothing
2      1003  2023-01-06            45      M   Bangalore         Food
3      1004  2023-01-06            31      F     Chennai         Books
4      1005  2023-01-07            29      M        Pune         Home

   product_name  quantity  unit_price   revenue  rating  returned
0    Smartphone         1     15000.0   15000.0     4.2     False
1         Shirt         2       800.0    1600.0     3.8     False
2         Pizza         3       450.0    1350.0     4.5     False
3         Novel         1       350.0     350.0     4.0     False
4      Cushion         4       275.0    1100.0     3.9     False

# Calculate sample statistics for revenue
sample_mean = df['revenue'].mean()
sample_std = df['revenue'].std()
sample_size = len(df)

# Display key statistics
print(f"Sample mean: ₹{sample_mean:.2f}")
print(f"Sample std: ₹{sample_std:.2f}")
print(f"Sample size: {sample_size}")

Sample mean: ₹3247.85
Sample std: ₹4156.73
Sample size: 1000

# Set up hypothesis test parameters
claimed_mean = 1500  # Flipkart's claim
alpha = 0.05  # 5% significance level

# State the hypotheses clearly
print("H₀: μ = ₹1,500 (Flipkart's claim is correct)")
print("H₁: μ ≠ ₹1,500 (Flipkart's claim is incorrect)")
print(f"Significance level: {alpha}")
print(f"Test type: Two-tailed")

H₀: μ = ₹1,500 (Flipkart's claim is correct)
H₁: μ ≠ ₹1,500 (Flipkart's claim is incorrect)
Significance level: 0.05
Test type: Two-tailed

# Perform one-sample t-test using scipy
t_statistic, p_value = stats.ttest_1samp(df['revenue'], claimed_mean)

# Calculate degrees of freedom
df_freedom = sample_size - 1

# Display test results
print(f"t-statistic: {t_statistic:.4f}")
print(f"p-value: {p_value:.6f}")
print(f"degrees of freedom: {df_freedom}")

t-statistic: 13.3104
p-value: 0.000000
degrees of freedom: 999

# Make statistical decision
if p_value < alpha:
    decision = "Reject H₀"
    conclusion = "significant"
else:
    decision = "Fail to reject H₀" 
    conclusion = "not significant"

print(f"Decision: {decision}")
print(f"Conclusion: The difference is {conclusion}")
print(f"Business impact: Flipkart's claim appears incorrect")

Decision: Reject H₀
Conclusion: The difference is significant
Business impact: Flipkart's claim appears incorrect

Our t-statistic (13.31) falls far outside the critical region, providing strong evidence against the null hypothesis

The scatter plot shows where our test statistic lands relative to the critical values. Our purple dot is way off the chart - that's how extreme our result is. The red dots mark the critical region boundaries at ±1.96 for α=0.05.

This visualization makes it crystal clear: our observed difference isn't just a random fluctuation. The average order value is genuinely different from ₹1,500, with statistical confidence exceeding 99.9%.

Two-Sample Testing: Comparing Groups

The scenario: Swiggy's marketing team needs to know if male and female customers have different average order values. This determines whether they should create gender-specific promotional campaigns or stick with universal messaging.

# Separate revenue by gender
male_revenue = df[df['gender'] == 'M']['revenue']
female_revenue = df[df['gender'] == 'F']['revenue']

# Calculate group statistics
male_mean = male_revenue.mean()
female_mean = female_revenue.mean()

print(f"Male avg order: ₹{male_mean:.2f}")
print(f"Female avg order: ₹{female_mean:.2f}")
print(f"Difference: ₹{male_mean - female_mean:.2f}")

Male avg order: ₹3,285.67
Female avg order: ₹3,210.03
Difference: ₹75.64

# Perform independent two-sample t-test
t_stat, p_val = stats.ttest_ind(male_revenue, female_revenue)

# Set significance level
alpha = 0.05

print(f"Two-sample t-test results:")
print(f"t-statistic: {t_stat:.4f}")
print(f"p-value: {p_val:.4f}")
print(f"Alpha: {alpha}")

Two-sample t-test results:
t-statistic: 0.3842
p-value: 0.7011
Alpha: 0.05

The difference between male and female order values appears minimal and statistically insignificant

The bar chart confirms what our statistical test revealed. While males spend slightly more, the difference is too small to be meaningful. From a business perspective, Swiggy shouldn't waste resources on gender-specific campaigns based on this evidence.

But here's what trips up most analysts: they see the ₹75 difference and think it's actionable. Statistical testing saves you from making expensive decisions based on random noise. The p-value of 0.70 means there's a 70% chance this difference occurred purely by random sampling.

Statistical Power and Effect Size

Here's where 90% of data scientists get tripped up. You can have a statistically significant result that's practically meaningless, or a meaningful difference that doesn't reach statistical significance due to small sample size.

# Calculate Cohen's d for effect size
def cohens_d(group1, group2):
    # Calculate pooled standard deviation
    pooled_std = np.sqrt(((len(group1)-1)*group1.std()**2 + (len(group2)-1)*group2.std()**2) / (len(group1)+len(group2)-2))
    # Return effect size
    return (group1.mean() - group2.mean()) / pooled_std

effect_size = cohens_d(male_revenue, female_revenue)
print(f"Cohen's d (effect size): {effect_size:.4f}")

Cohen's d (effect size): 0.0192

Aim for 80% statistical power to reliably detect meaningful differences when they exist

The doughnut chart shows the standard 80% power target. Statistical power is your test's ability to detect a real effect when one exists. Low power means you might miss important business insights due to insufficient sample size.

Power depends on four factors: effect size, sample size, significance level, and population variability. You can't control population variability, but you can calculate required sample sizes before collecting data. This prevents the frustrating situation where you collect data but can't draw confident conclusions.

Quiz