Correlation Analysis

import pandas as pd      # pandas: the main Python library for working with data tables — like Excel, but in code
import numpy as np       # numpy: a library for doing maths on lists of numbers quickly
from scipy import stats  # scipy: a science and statistics library — gives us the correlation tests

# 12 customers — their order value (£) and star rating (1–5)
# Notice customer 12 spent £4,200 — a clear outlier compared to everyone else
df = pd.DataFrame({
    'customer_id':  range(1, 13),
    'order_value':  [45, 82, 31, 120, 67, 95, 54, 38, 110, 73, 88, 4200],  # customer 12 is the outlier
    'star_rating':  [3,  4,  2,  5,   3,  4,  3,  2,  5,   4,  4,  3  ]   # ratings are 1–5 stars
})

print("Our data:")
print(df[['customer_id', 'order_value', 'star_rating']].to_string(index=False))
print()

# --- PEARSON CORRELATION ---
# Pearson looks at the actual numbers. That £4,200 will pull the result hard.
# stats.pearsonr() returns two numbers: (r, p_value)
# r = the correlation score  |  p_value = how likely this result is just random chance
pearson_r, pearson_p = stats.pearsonr(df['order_value'], df['star_rating'])

# --- SPEARMAN CORRELATION ---
# Spearman converts values to ranks first: cheapest order = rank 1, most expensive = rank 12
# Then it measures whether the rank orders match
# The £4,200 outlier becomes "rank 12" — a big number, but not impossibly big
spearman_r, spearman_p = stats.spearmanr(df['order_value'], df['star_rating'])

print(f"Pearson  r = {pearson_r:.3f}   p = {pearson_p:.4f}")
print(f"Spearman r = {spearman_r:.3f}   p = {spearman_p:.4f}")
print()
print("Without the outlier (customer 12 removed):")
df_no_outlier = df[df['customer_id'] != 12]   # remove the outlier row to compare
p_r2, _ = stats.pearsonr(df_no_outlier['order_value'], df_no_outlier['star_rating'])
s_r2, _ = stats.spearmanr(df_no_outlier['order_value'], df_no_outlier['star_rating'])
print(f"Pearson  r = {p_r2:.3f}   (was {pearson_r:.3f}  — changed a lot!)")
print(f"Spearman r = {s_r2:.3f}   (was {spearman_r:.3f} — barely changed)")

What just happened?

pandas is our data table tool — it holds the customer data in rows and columns, just like a spreadsheet. We use it to store the data and filter out the outlier row with a simple condition.

scipy is a science and statistics library. stats.pearsonr() calculates Pearson correlation and returns both the r value and the p-value (explained below). stats.spearmanr() does the same for Spearman — it quietly converts your values to ranks internally before computing.

This output is a perfect demonstration of why outliers matter. With the £4,200 order included, Pearson says r=0.143 — almost zero, no relationship. But that's a lie caused by one extreme customer distorting the straight-line calculation. Remove them and Pearson jumps to 0.942. Spearman barely flinches — it was already giving the right answer (0.745) because it works on ranks, not raw numbers. One outlier can completely mislead Pearson. Spearman shrugs it off.

The p-value in plain English

Imagine you flipped a coin 10 times and got 8 heads. Is the coin biased, or did you just get lucky? The p-value answers exactly this kind of question.

A p-value of 0.05 means: "If there were truly no relationship between these two variables, there's only a 5% chance I'd see a result this strong just by random luck." Below 0.05, analysts say the result is statistically significant — it's probably a real pattern, not noise.

A high p-value (say, 0.65) means: "This result could easily happen by chance even if nothing real is going on." Don't trust it.

import pandas as pd      # pandas: our data table library — storing data, selecting columns
import numpy as np       # numpy: fast maths on lists of numbers
from scipy import stats  # scipy: statistics library — all three correlation methods live here

# A customer behaviour dataset — mix of normal data and some skewed columns
df = pd.DataFrame({
    'days_since_signup': [120, 340, 45, 280, 190, 410, 60, 155, 320, 85, 230, 175],
    'total_orders':      [3,   12,  1,  9,   6,   15,  2,  5,   11,  2,  8,   6  ],
    'avg_order_value':   [45,  82,  31, 120,  67,  95,  54, 38, 110,  73, 88,  4200],  # outlier in last row
    'support_tickets':   [1,   3,   0,  2,   1,   4,   0,  1,   3,   0,  2,   1  ],
    'star_rating':       [3,   4,   2,  5,   3,   4,   3,  2,   5,   4,  4,   3  ]
})

# For each pair of columns we care about, run all three correlation methods
pairs = [
    ('total_orders',     'days_since_signup'),   # do older customers order more?
    ('avg_order_value',  'star_rating'),          # do big spenders rate higher? (outlier pair!)
    ('total_orders',     'support_tickets'),      # do frequent buyers raise more issues?
]

print(f"{'Pair':<40} {'Pearson':>8} {'Spearman':>9} {'Kendall':>8}  Recommendation")
print("-" * 85)

for col_a, col_b in pairs:
    p_r,  _ = stats.pearsonr(df[col_a], df[col_b])     # Pearson: sensitive to outliers, linear only
    sp_r, _ = stats.spearmanr(df[col_a], df[col_b])    # Spearman: rank-based, handles outliers
    k_r,  _ = stats.kendalltau(df[col_a], df[col_b])   # Kendall's tau: agreement-based, good for small n

    # If Pearson and Spearman disagree by more than 0.3, outliers are likely the cause
    flag = " ← outlier suspected" if abs(p_r - sp_r) > 0.3 else ""

    label = f"{col_a} × {col_b}"
    print(f"{label:<40} {p_r:>8.3f} {sp_r:>9.3f} {k_r:>8.3f}  {flag}")

What just happened?

scipy provides all three correlation functions in its stats submodule. stats.pearsonr() measures linear relationships directly on the raw numbers. stats.spearmanr() converts values to ranks first, then measures whether the ranks agree. stats.kendalltau() counts concordant pairs (both go up together) vs discordant pairs (one goes up, the other goes down) and divides them — a slightly different concept but a similar result.

The automatic outlier flag is the practical gem here: when Pearson and Spearman disagree by more than 0.3, it's almost always because an outlier is distorting Pearson. This check takes one line and saves you from reporting a misleading correlation.

The first and third pairs (total_orders × days_since_signup, and total_orders × support_tickets) show all three methods agreeing — strong positive relationships, no outlier issue. The middle pair (avg_order_value × star_rating) is the flagged one: Pearson says 0.143 (almost nothing), Spearman says 0.745 (strong). That gap of 0.60 is the outlier customer screaming for attention.

import pandas as pd      # pandas: data table library — selecting columns and checking skewness
import numpy as np       # numpy: maths library — abs() for comparing values
from scipy import stats  # scipy: statistics library — correlation functions and their p-values

def smart_correlation_report(dataframe, col_a, col_b):
    """
    Automatically picks the right correlation method based on the data,
    returns a plain-English summary.
    """
    a = dataframe[col_a].dropna()
    b = dataframe[col_b].dropna()

    # Decide which method to use:
    # - If either column is heavily skewed (|skew| > 1) → Spearman is safer
    # - Otherwise → Pearson is fine
    skew_a = abs(a.skew())   # .skew() from pandas tells us how lopsided the distribution is
    skew_b = abs(b.skew())

    if skew_a > 1 or skew_b > 1:
        method = 'Spearman'
        r, p   = stats.spearmanr(a, b)
        reason = f"(used because skewness is high: {col_a}={skew_a:.1f}, {col_b}={skew_b:.1f})"
    else:
        method = 'Pearson'
        r, p   = stats.pearsonr(a, b)
        reason = f"(used because both columns are roughly symmetric)"

    # Plain-English interpretation of the r value
    if abs(r) >= 0.7:     strength = "strong"
    elif abs(r) >= 0.4:   strength = "moderate"
    else:                  strength = "weak"

    direction = "positive" if r > 0 else "negative"
    sig       = "significant (p<0.05)" if p < 0.05 else "NOT significant (p≥0.05)"

    print(f"{'='*52}")
    print(f"  {col_a}  ×  {col_b}")
    print(f"{'='*52}")
    print(f"  Method:  {method} {reason}")
    print(f"  r = {r:.3f}  |  p = {p:.4f}  |  {sig}")
    print(f"  Finding: {strength} {direction} correlation")
    print()

# Run the smart report on three key pairs
smart_correlation_report(df, 'total_orders',    'days_since_signup')
smart_correlation_report(df, 'avg_order_value', 'star_rating')
smart_correlation_report(df, 'total_orders',    'support_tickets')

What just happened?

pandas' .skew() method is the key decision-maker here — it tells us whether a column is lopsided. If skewness is above 1 (highly lopsided), we automatically switch to Spearman because Pearson assumes a roughly symmetric, bell-shaped distribution. This turns method selection from a manual judgement call into an automatic check in your code.

scipy's correlation functions return both the correlation value and the p-value in a single call. We use both — the r value to describe strength and direction, the p-value to decide whether to trust it.

The function caught the avg_order_value column's extreme skewness (3.4) and automatically used Spearman — giving the correct r=0.745 instead of the misleading Pearson 0.143. This is the kind of automated safeguard that separates a robust analysis from a fragile one.

Teacher's Note

Correlation does not mean causation. This phrase gets repeated so often it stops landing. Here's a concrete version: ice cream sales and drowning deaths are strongly positively correlated. Does ice cream cause drowning? No. Both spike in summer when it's hot. The real driver — temperature — is hidden. A correlation just says two things move together. It says nothing about why.

The practical rule: when you find a strong correlation, your next question should always be "what third variable could explain this?" That discipline alone will save you from embarrassing conclusions in front of stakeholders.

Up Next · Lesson 20

Covariance Analysis

Correlation's lesser-known cousin — understand what covariance actually measures, why it's harder to interpret, and when it matters more than correlation.