Covariance Analysis

The formula in plain words: Correlation = Covariance ÷ (standard deviation of A × standard deviation of B). Dividing by the standard deviations "standardises" the covariance — it removes the effect of scale and gives you a number that always lands between −1 and +1. That's the only difference.

import pandas as pd      # pandas: our main data table tool — like a spreadsheet in Python
import numpy as np       # numpy: fast maths library — we'll use it to manually compute covariance too

# Gym membership data — 10 members
df = pd.DataFrame({
    'member_id':        range(1, 11),
    'age':              [24, 35, 42, 28, 51, 33, 47, 29, 38, 55],
    'visits_per_month': [18, 12, 8,  15, 5,  14, 7,  16, 10, 4 ],
    'pt_spend_monthly': [0,  40, 80, 20, 120, 30, 90, 10, 50, 150]  # personal training spend in £
})

print("Our gym member data:")
print(df[['member_id','age','visits_per_month','pt_spend_monthly']].to_string(index=False))
print()

# pandas .cov() computes the full covariance matrix — every column vs every other column
# Think of it like a correlation matrix, but the numbers are in "raw" units rather than −1 to +1
cov_matrix = df[['age','visits_per_month','pt_spend_monthly']].cov()
print("=== COVARIANCE MATRIX ===")
print(cov_matrix.round(2))
print()

# Manual covariance between visits and pt_spend — to show exactly what's being calculated
# Covariance = average of (how far each A is from A's mean) × (how far each B is from B's mean)
visits_mean   = df['visits_per_month'].mean()
pt_spend_mean = df['pt_spend_monthly'].mean()

# For each member, multiply their deviation from the visits mean by their deviation from the spend mean
deviations = (df['visits_per_month'] - visits_mean) * (df['pt_spend_monthly'] - pt_spend_mean)

# The covariance is the average of all those products
# ddof=1 means we divide by (n-1) not n — the standard correction for sample data
manual_cov = deviations.sum() / (len(df) - 1)
print(f"Manual covariance (visits × pt_spend): {manual_cov:.2f}")
print(f"pandas .cov() result:                  {df['visits_per_month'].cov(df['pt_spend_monthly']):.2f}")

What just happened?

pandas is our data table library. The .cov() method computes a covariance matrix — a table where every cell shows how two columns vary together. The diagonal cells (age with age, visits with visits) are just the variance of each column — how spread out it is on its own. The off-diagonal cells tell us how pairs of columns move together.

numpy is our fast maths library. We used it here indirectly — pandas' .cov() uses numpy under the hood for the actual calculations.

Now let's read the output. The covariance between visits_per_month and pt_spend_monthly is −164.44. That's negative — members who visit more tend to spend less on personal training. Makes sense: frequent visitors are gym regulars who don't need a PT. The covariance between age and pt_spend_monthly is +335.56 — older members spend more on personal training. But notice: is 335.56 a "big" number? Without knowing the units, it's hard to say. That's the limitation of covariance — which is exactly why we convert to correlation next.

import pandas as pd      # pandas: data table library — .cov(), .corr(), and column operations
import numpy as np       # numpy: maths library — manual formula demonstration

# Show how covariance changes with scale but correlation doesn't
# Original data: PT spend in pounds (£)
spend_pounds = df['pt_spend_monthly']

# Same data but converted to pence (×100) — same real-world relationship, different units
spend_pence  = df['pt_spend_monthly'] * 100   # multiply every value by 100

# --- COVARIANCE: changes when units change ---
cov_pounds = df['age'].cov(spend_pounds)      # covariance with spend in £
cov_pence  = df['age'].cov(spend_pence)       # covariance with spend in pence

print("=== THE SCALE PROBLEM ===")
print(f"Covariance (age × PT spend in £):     {cov_pounds:.2f}")
print(f"Covariance (age × PT spend in pence): {cov_pence:.2f}   ← 100× bigger!")
print(f"Ratio: {cov_pence/cov_pounds:.0f}×  (same relationship, wildly different numbers)")
print()

# --- CORRELATION: stays exactly the same regardless of units ---
from scipy import stats   # scipy: statistics library — pearsonr for correlation with p-value

corr_pounds, p_pounds = stats.pearsonr(df['age'], spend_pounds)
corr_pence,  p_pence  = stats.pearsonr(df['age'], spend_pence)

print("=== CORRELATION IS SCALE-FREE ===")
print(f"Correlation (age × PT spend in £):     r = {corr_pounds:.4f}")
print(f"Correlation (age × PT spend in pence): r = {corr_pence:.4f}  ← identical!")
print()

# Now derive correlation from covariance manually — so you can see the relationship
# Formula: r = cov(A,B) / (std(A) × std(B))
std_age   = df['age'].std()
std_spend = spend_pounds.std()
r_manual  = cov_pounds / (std_age * std_spend)   # divide by both standard deviations
print(f"Manual derivation: {cov_pounds:.2f} / ({std_age:.2f} × {std_spend:.2f}) = {r_manual:.4f}")
print(f"This matches pearsonr: {corr_pounds:.4f}  ✓")

What just happened?

pandas is doing the covariance calculations via .cov(). When we multiply PT spend by 100 (converting pounds to pence), the covariance jumps from 335 to 33,555 — 100 times bigger. The data is identical; only the label on the ruler changed.

scipy's stats.pearsonr() shows the correlation stays at exactly 0.9723 in both cases. Because dividing by the standard deviations cancels out the unit change.

The manual derivation at the bottom is the most important part: correlation = covariance ÷ (std_A × std_B). This isn't magic — it's simple arithmetic. Seeing this once makes the relationship between covariance and correlation unforgettable. Covariance gives you the direction and raw magnitude. Dividing by the standard deviations standardises it into a −1 to +1 scale you can always interpret.

import pandas as pd      # pandas: data table library — .cov(), .var(), and column operations
import numpy as np       # numpy: maths library — matrix operations for portfolio variance

# Monthly revenue data (£000s) for two gym revenue streams over 12 months
# Membership fees are stable; PT revenue is more volatile
revenue = pd.DataFrame({
    'month':        range(1, 13),
    'membership':   [42, 43, 41, 44, 45, 43, 42, 44, 46, 45, 43, 44],   # fairly steady
    'pt_revenue':   [18, 22, 14, 28, 32, 19, 15, 25, 35, 30, 17, 21]    # more volatile
})

# Variance of each revenue stream individually
# Variance = average of squared deviations from the mean — a measure of "bumpiness"
var_membership = revenue['membership'].var()   # pandas .var() — sample variance (divides by n-1)
var_pt         = revenue['pt_revenue'].var()

print(f"Variance — membership fees:    {var_membership:.2f}  (lower = more stable)")
print(f"Variance — PT revenue:         {var_pt:.2f}  (higher = more volatile)")
print()

# Covariance between the two streams
cov_streams = revenue['membership'].cov(revenue['pt_revenue'])
print(f"Covariance between streams:    {cov_streams:.2f}")
print()

# Portfolio variance formula:
# If you combine two revenue streams (50/50 split), the combined variance is:
# Var_portfolio = w1² × Var1  +  w2² × Var2  +  2 × w1 × w2 × Cov(1,2)
# where w1 and w2 are the weights (both 0.5 for a 50/50 split)
w1, w2 = 0.5, 0.5
var_portfolio = (w1**2 * var_membership) + (w2**2 * var_pt) + (2 * w1 * w2 * cov_streams)

print(f"Portfolio variance (50/50 mix): {var_portfolio:.2f}")
print(f"Portfolio std dev (volatility):  £{var_portfolio**0.5:.2f}k per month")
print()

# Compare: what if PT revenue was negatively correlated with membership?
# Simulate an inverse PT stream
revenue['pt_inverse'] = 42 - (revenue['pt_revenue'] - revenue['pt_revenue'].mean())
cov_inverse = revenue['membership'].cov(revenue['pt_inverse'])
var_portfolio_inverse = (w1**2 * var_membership) + (w2**2 * revenue['pt_inverse'].var()) + (2 * w1 * w2 * cov_inverse)

print(f"If PT revenue moved OPPOSITE to membership:")
print(f"Portfolio variance would be:    {var_portfolio_inverse:.2f}  ← much lower!")
print(f"Portfolio std dev:              £{var_portfolio_inverse**0.5:.2f}k per month  (less risk)")

What just happened?

pandas' .var() computes the variance of a column — how spread out the values are. PT revenue (variance 42.08) is 20× more "bumpy" than membership fees (2.08). That's visible just from glancing at the numbers in the dataset.

numpy powers the arithmetic behind the portfolio variance formula. The formula itself — w1² × Var1 + w2² × Var2 + 2 × w1 × w2 × Cov — is the reason covariance can't be replaced by correlation here. The formula needs the raw covariance number, not a −1 to +1 score. You need the actual size of how the two streams move together to compute a real variance in £ terms.

The key takeaway: when PT revenue moves in the same direction as membership, combining them gives a portfolio volatility of £3.72k/month. When they move in opposite directions, volatility drops to £2.95k/month — nearly 20% less risky. This is diversification in a formula, and covariance is what makes it computable.

import pandas as pd      # pandas: data library — .cov() and .corr() for the two matrices
import numpy as np       # numpy: maths library — standard import

numeric_cols = ['age', 'visits_per_month', 'pt_spend_monthly']

# The covariance matrix — raw, units-dependent, hard to compare across pairs
cov_mat  = df[numeric_cols].cov()

# The correlation matrix — standardised, always −1 to +1, easy to compare
corr_mat = df[numeric_cols].corr()   # pandas .corr() uses Pearson by default

print("=== COVARIANCE MATRIX (raw — unit-dependent) ===")
print(cov_mat.round(2))
print()
print("=== CORRELATION MATRIX (standardised — always −1 to +1) ===")
print(corr_mat.round(3))
print()

# Plain-English interpretation of each relationship
print("=== PLAIN-ENGLISH SUMMARY ===")
pairs = [
    ('age', 'visits_per_month'),
    ('age', 'pt_spend_monthly'),
    ('visits_per_month', 'pt_spend_monthly')
]

for a, b in pairs:
    cov_val  = cov_mat.loc[a, b]
    corr_val = corr_mat.loc[a, b]
    direction = "positive" if corr_val > 0 else "negative"
    strength  = "strong" if abs(corr_val) > 0.7 else "moderate" if abs(corr_val) > 0.4 else "weak"
    print(f"  {a} × {b}:")
    print(f"    Covariance = {cov_val:.2f}  |  Correlation = {corr_val:.3f}")
    print(f"    → {strength} {direction} relationship")
    print()

What just happened?

pandas makes both matrices trivial: .cov() and .corr() are called identically — the only difference is which number lands in each cell. The covariance matrix values range from 23 to 2,572 — impossible to compare at a glance. The correlation matrix values are all between −1 and +1, making every cell instantly readable.

The correlations here are remarkably strong — age vs visits is −0.999 (almost perfectly negative: the older the member, the fewer times they visit). That pattern was in the covariance matrix too (−47.78), but you couldn't tell it was nearly perfect without doing the mental maths to compare it against the other cells. Correlation makes that comparison instant.

Teacher's Note

In day-to-day EDA, you will use correlation far more than covariance. Correlation is easier to read, easier to explain to a non-technical stakeholder, and doesn't depend on units. If your manager asks "how related are these two things?", the answer is a correlation coefficient.

But covariance isn't just a stepping stone to correlation. It's the right tool whenever you need the size of how two things move together in real units — portfolio risk calculations, principal component analysis (coming in Lesson 39), and multivariate statistical models all need covariance, not just correlation. Learn both; know when to reach for each.

Up Next · Lesson 21

Feature Relationships

Go beyond pairwise numbers — learn to map out the full web of relationships between all your features and spot which ones are genuinely useful for prediction.