Categorical Exploration

import pandas as pd
import numpy as np

# Our fashion retailer orders dataset
df = pd.DataFrame({
    'order_id':       range(1, 15),
    'category':       ['Tops','Shoes','Tops','Dresses','Shoes','Tops',
                       'Accessories','Dresses','Shoes','Tops','Dresses',
                       'Accessories','Shoes','Tops'],
    'customer_tier':  ['Gold','Silver','Bronze','Gold','Silver','Bronze',
                       'Gold','Bronze','Silver','Gold','Bronze','Silver',
                       'Gold','Silver'],
    'return_reason':  ['Wrong size','No reason','Wrong size','Damaged',
                       'Wrong size','No reason','No reason','Damaged',
                       'Wrong size','Wrong size','No reason','Damaged',
                       'Wrong size','No reason'],
    'satisfaction':   ['Happy','Neutral','Unhappy','Neutral','Happy',
                       'Happy','Happy','Unhappy','Neutral','Happy',
                       'Neutral','Unhappy','Happy','Neutral'],
    'order_value':    [45, 89, 32, 120, 67, 28, 55, 98, 74, 41, 110, 63, 82, 37]
})

print(df.to_string(index=False))

What just happened?

pandas is our data table library — it stores the data in rows and columns, just like a spreadsheet. We used pd.DataFrame() to create the table, and .to_string(index=False) to print it cleanly without the row numbers on the left. You can see four text columns and one number column — a typical mix in real retail data.

cat_cols = ['category', 'customer_tier', 'return_reason', 'satisfaction']

for col in cat_cols:
    # .nunique() counts how many DIFFERENT values exist in this column
    n_unique = df[col].nunique()

    # .unique() shows you exactly what those values are
    unique_vals = list(df[col].unique())

    print(f"{col}: {n_unique} unique values → {unique_vals}")

What just happened?

pandas' .nunique() counts the number of distinct values in a column. .unique() returns the actual list of those values. Together they give you a complete picture of what's in each column. All four columns have low cardinality (3–4 values) — easy to encode. A column with 500 unique values would need a very different strategy.

for col in cat_cols:
    print(f"--- {col} ---")

    # .value_counts() counts how many times each category appears
    # normalize=True converts counts to percentages (0.0 to 1.0)
    counts = df[col].value_counts()
    pcts   = df[col].value_counts(normalize=True) * 100  # multiply by 100 for %

    for category in counts.index:
        bar = '█' * counts[category]   # simple visual bar — one block per order
        print(f"  {category:<15} {counts[category]:>2} orders  ({pcts[category]:.0f}%)  {bar}")
    print()

What just happened?

pandas' .value_counts() is the go-to method for categorical columns — it counts how many times each value appears, sorted from most to least common. Adding normalize=True converts those counts to proportions so you can see percentages.

Good news: the distribution looks fairly balanced here — no category dominates. "Wrong size" is the most common return reason at 43%, which makes immediate business sense. "Unhappy" is the least common satisfaction rating at 21% — worth flagging but not alarming.

RARE_THRESHOLD = 0.15   # any category below 15% of rows is "rare"

print("=== RARE CATEGORY CHECK ===\n")

for col in cat_cols:
    # Get proportions for each category (0.0 to 1.0)
    proportions = df[col].value_counts(normalize=True)

    # Find the ones below our threshold
    rare = proportions[proportions < RARE_THRESHOLD]

    if len(rare) > 0:
        print(f"  ⚠  {col}: rare categories found:")
        for cat, pct in rare.items():
            print(f"       '{cat}' appears in only {pct*100:.1f}% of rows")
    else:
        print(f"  ✓  {col}: no rare categories")

All clear here — but in real data, this often catches problems

In a real dataset with thousands of rows, you might find a product category that appears just twice. When that category gets one-hot encoded into a column, you'd have a column that's 99.8% zeros and 0.2% ones — almost no signal for the model. The fix: merge rare categories into an "Other" bucket before encoding.

# Define the correct order manually — lowest to highest
satisfaction_order = {'Unhappy': 0, 'Neutral': 1, 'Happy': 2}

# .map() applies this dictionary to every row in the column
# 'Unhappy' becomes 0, 'Neutral' becomes 1, 'Happy' becomes 2
df['satisfaction_encoded'] = df['satisfaction'].map(satisfaction_order)

# Show the result side by side
print(df[['satisfaction', 'satisfaction_encoded']].drop_duplicates().sort_values('satisfaction_encoded'))

What just happened?

pandas' .map() takes a dictionary and replaces every value in the column using it as a lookup table. We defined the order ourselves — Unhappy=0, Neutral=1, Happy=2 — so the numbers respect the real-world meaning. The model can now understand that 2 (Happy) is better than 0 (Unhappy), which is exactly the relationship we want it to learn.

# pd.get_dummies() creates one new column per category
# prefix='cat' keeps column names readable: cat_Tops, cat_Shoes, etc.
# drop_first=True removes one column to avoid a maths problem called multicollinearity
#   (if you know it's not Tops/Shoes/Dresses, it must be Accessories — so that column is redundant)
dummies = pd.get_dummies(df['category'], prefix='cat', drop_first=True)

print("One-hot encoded columns:")
print(dummies)

What just happened?

pandas' pd.get_dummies() automatically finds every unique value in the column and creates a True/False column for each one. We used drop_first=True to remove one column (Accessories) — because if all three other columns are False, the order must be Accessories. Keeping all four would give the model redundant information.

Row 0 is a Top: cat_Tops = True, the others are False. Row 3 is a Dress: cat_Dresses = True, the others are False. Row 6 is Accessories (the dropped category): all three are False. The model can figure that out on its own.

# .groupby() splits the data by category, then calculates the mean order value per group
# This shows us: does a customer's category affect how much they spend?
print("--- Average order value by product category ---")
print(df.groupby('category')['order_value'].mean().sort_values(ascending=False).round(1))
print()

# Now check satisfaction — do unhappy customers spend more or less?
print("--- Average order value by satisfaction level ---")
# We use the encoded version so it sorts correctly: Unhappy < Neutral < Happy
print(df.groupby('satisfaction')['order_value'].mean())
print()

# Return reason breakdown: what % of each return reason comes from each category?
print("--- Return reasons by product category ---")
# pd.crosstab() counts how many times each combination of two categories appears
print(pd.crosstab(df['return_reason'], df['category']))

What just happened?

pandas' .groupby() splits the data into groups and computes a statistic for each. pd.crosstab() counts how many rows share each combination of two categorical columns — like a tally chart.

Three genuinely interesting findings pop out of this:

① Dresses have the highest average order value (£109) — the premium category. Tops are the cheapest (£37).

② Unhappy customers actually spend more on average (£84) than Happy ones (£55). This is a counterintuitive but important finding — higher-value orders may lead to higher expectations and more disappointment.

③ "Wrong size" returns are almost exclusively Shoes (4 out of 6). That's an actionable insight for the returns model — and for the product team who might need better sizing guides on footwear.

Teacher's Note

The biggest mistake people make with categorical columns is jumping straight to encoding without exploring first. Encoding before exploring means you might encode a column that has no relationship with your target at all — wasting columns and confusing your model.

Always follow this order: count → frequency → rare check → ordinal/nominal → connect to outcome → then encode. Five minutes of exploration saves hours of debugging a model that silently underperforms because of a badly encoded feature.

Up Next · Lesson 23

Numerical Exploration

The deep dive into numeric columns — distributions, percentiles, outlier zones, and how to build a complete numeric profile you can hand to any data scientist.