Skewness & Kurtosis

import pandas as pd          # pandas: Python's core data table library — we use it to build and analyse our DataFrame
import numpy as np           # numpy: numerical Python library — we need it for np.nan and fast array operations

# Build a realistic dataset of monthly flat rentals (£/month) across three London zones
data = {
    'central':  [3200, 3450, 3100, 2950, 3600, 3300, 3150, 6800, 7200, 3250],  # central London — note the two expensive outliers
    'mid_zone': [1800, 1750, 1900, 1850, 1820, 1780, 1950, 1760, 1840, 1890],  # mid-zone — fairly tight cluster
    'outer':    [1100, 1050, 1200, 1080, 1120, 1090, 1060, 1150, 1070, 1130]   # outer boroughs — also clustered, lower rents
}

df = pd.DataFrame(data)      # turn the dictionary into a DataFrame for easy column-by-column analysis

# Calculate skewness for each zone
# pandas .skew() uses the standard unbiased Fisher-Pearson skewness formula
skewness = df.skew()         # returns a Series with one skewness value per column
print("=== SKEWNESS ===")
print(skewness.round(3))     # round to 3 decimal places for readability
print()

# Calculate kurtosis for each zone
# IMPORTANT: pandas .kurt() returns EXCESS kurtosis (normal distribution = 0, not 3)
kurtosis = df.kurt()         # returns a Series with one kurtosis value per column
print("=== EXCESS KURTOSIS ===")
print(kurtosis.round(3))

What just happened?

pandas is Python's core data table library. Here we used two of its statistical methods: .skew() and .kurt(). When called on a DataFrame (no column specified), both methods run across every column and return a Series — one value per column. That's the "apply across all columns" default behaviour.

numpy is Python's numerical powerhouse. We imported it here as standard practice — it's used later in the lesson and is almost always present alongside pandas in EDA work.

The central zone has a skewness of 1.847 — that's strongly positive. The two expensive flats at £6,800 and £7,200 are pulling the right tail hard. A model trained on this raw data would overestimate how common high rents are.

The mid_zone is close to symmetric (0.213), and outer is almost perfectly symmetric (-0.021). These zones are much safer to feed into a model as-is.

The kurtosis for central is 3.412 — strongly leptokurtic. Those two extreme values create exactly the fat tail behaviour kurtosis is designed to catch. Mid-zone and outer both have negative kurtosis (platykurtic) — their values are tightly packed, with even fewer extremes than you'd expect from a normal distribution.

import pandas as pd                           # pandas: core data library — DataFrame and column access
import numpy as np                            # numpy: numerical computing library — array handling
from scipy import stats                       # scipy.stats: scientific computing statistics module — contains formal hypothesis tests for skewness and kurtosis

# Recreate the central London rent data
central_rents = np.array([3200, 3450, 3100, 2950, 3600, 3300, 3150, 6800, 7200, 3250])  # 10 values — small sample but realistic

# scipy's skewtest: tests whether skewness is significantly different from zero
# Returns a statistic and a two-tailed p-value
# Null hypothesis: the data comes from a symmetric distribution (skewness = 0)
skew_stat, skew_p = stats.skewtest(central_rents)   # unpack into statistic and p-value
print(f"Skew test statistic: {skew_stat:.3f}")       # size of the skewness signal
print(f"Skew test p-value:   {skew_p:.4f}")          # probability of seeing this skewness by chance

print()                                              # blank line for readability

# scipy's kurtosistest: tests whether excess kurtosis is significantly different from zero
# Null hypothesis: the data has normal-distribution tail behaviour (excess kurtosis = 0)
kurt_stat, kurt_p = stats.kurtosistest(central_rents)  # same structure as skewtest
print(f"Kurtosis test statistic: {kurt_stat:.3f}")
print(f"Kurtosis test p-value:   {kurt_p:.4f}")

print()
# Interpret results clearly for your stakeholder
alpha = 0.05                                            # standard significance threshold
if skew_p < alpha:
    print("Verdict: Statistically significant skewness detected — transformation recommended.")  # flag it
else:
    print("Verdict: No statistically significant skewness at the 0.05 level.")  # safe to proceed

What just happened?

scipy is Python's scientific computing library. It sits on top of numpy and provides statistical tests, distributions, signal processing, and optimisation routines. We specifically imported scipy.stats, which is the module dedicated to statistical functions. It's the go-to library when pandas' built-in statistics aren't enough and you need formal hypothesis testing.

stats.skewtest() runs the D'Agostino skewness test. It asks: "Given this sample size and this skewness value, what's the probability we'd see this result if the true population were actually symmetric?" A p-value of 0.0413 means there's only a 4.1% chance — below our 5% threshold — so we reject the null and flag the skewness as real.

The kurtosis test result (p = 0.0287) confirms the fat tails are also statistically significant. With only 10 data points, passing both tests this convincingly is a strong signal. Those two £6,800+ flats aren't noise — they're structurally changing the distribution.

import pandas as pd    # pandas: our data table library — DataFrame creation and column operations
import numpy as np     # numpy: numerical library — used for array creation and NaN handling

# Rebuild the full rental dataset
data = {
    'central':  [3200, 3450, 3100, 2950, 3600, 3300, 3150, 6800, 7200, 3250],
    'mid_zone': [1800, 1750, 1900, 1850, 1820, 1780, 1950, 1760, 1840, 1890],
    'outer':    [1100, 1050, 1200, 1080, 1120, 1090, 1060, 1150, 1070, 1130]
}
df = pd.DataFrame(data)   # DataFrame gives us column-level .skew() and .kurt()

def shape_summary(dataframe):
    """Returns a DataFrame with skewness, kurtosis, and plain-English flags for each column."""
    results = []                                           # build a list of dicts, one per column
    for col in dataframe.select_dtypes(include='number').columns:  # only numeric columns
        skew_val = dataframe[col].skew()                   # pandas .skew() — Fisher-Pearson, unbiased
        kurt_val = dataframe[col].kurt()                   # pandas .kurt() — excess kurtosis (normal = 0)

        # Plain-English skewness label based on common thresholds
        if abs(skew_val) < 0.5:
            skew_flag = "Symmetric"
        elif abs(skew_val) < 1.0:
            skew_flag = "Moderate skew"
        else:
            skew_flag = "HIGH SKEW — consider transform"   # flag for the data scientist

        # Plain-English kurtosis label
        if abs(kurt_val) < 1.0:
            kurt_flag = "Normal tails"
        elif kurt_val > 1.0:
            kurt_flag = "Fat tails (leptokurtic)"          # more extreme values than expected
        else:
            kurt_flag = "Thin tails (platykurtic)"         # fewer extreme values than expected

        results.append({
            'column':    col,
            'skewness':  round(skew_val, 3),
            'kurtosis':  round(kurt_val, 3),
            'skew_flag': skew_flag,
            'kurt_flag': kurt_flag
        })

    return pd.DataFrame(results).set_index('column')      # column name as row index for readability

summary = shape_summary(df)    # run it
print(summary.to_string())     # print without truncation

What just happened?

pandas is doing the heavy lifting here. .select_dtypes(include='number') is a handy method that filters a DataFrame to only numeric columns — useful when your real-world datasets have mixed text and numbers. .skew() and .kurt() are called per column inside the loop, then the results are assembled into a new summary DataFrame using pd.DataFrame(results).

numpy is imported here as part of the standard EDA toolkit. In this function it's not directly called, but it underpins the pandas calculations under the hood.

The output is the kind of table you'd actually put in a project notebook or attach to a Slack message for your lead. At a glance: central zone needs a log transformation before modelling, the other two are fine as-is. That's a decision made in seconds, not an hour of squinting at histograms.

Common fixes for high skewness:

• Log transform — np.log1p(x) — flattens right-skewed data dramatically. The go-to for income, prices, counts.
• Square root transform — np.sqrt(x) — gentler than log, good for moderate skew.
• Box-Cox transform — scipy.stats.boxcox(x) — finds the optimal power transformation automatically.
• Winsorization — cap extreme values at a percentile threshold instead of removing them.

Teacher's Note

A quick watch-out: sample size matters a lot for skewness and kurtosis. With fewer than 50 data points, these measures are noisy — a skewness of 0.8 from 20 rows might be meaningless. Always check your sample size before drawing conclusions. As a rough rule: treat skewness as meaningful at n ≥ 30 and kurtosis at n ≥ 50. The scipy hypothesis tests help here because they bake sample size into the p-value automatically.

Up Next · Lesson 13

Data Transformation

Log transforms, Box-Cox, scaling, and encoding — the toolkit for turning messy raw data into model-ready features.