Transformations for Skewed Data

Skewed columns punish linear models, inflate error metrics, and make distance-based algorithms unreliable. The fix isn't magic — it's a handful of mathematical transformations that compress the tail and bring the distribution closer to symmetry.

Skewness Hurts Models — Here's the Damage

Skewness is everywhere in real data. House prices, loan amounts, transaction values, website session lengths, salaries — nearly every financial or behavioural column in a real dataset has a right-skewed distribution. A handful of extreme values in the upper tail create three compounding problems:

The Four Transformations You Actually Use

There are dozens of mathematical transformations — but in practice, four do the heavy lifting. Here is what each one does and when to reach for it:

Step 1 — Measuring Skewness Before Transforming

The scenario: You're a data scientist at a property analytics firm. Your housing dataset has several numerical columns and your manager has complained that the regression model's residuals are non-normal — a classic symptom of skewed inputs. Before touching anything, you want to measure which columns are actually skewed and by how much, so you can prioritise which ones need treatment.

# Import libraries
import pandas as pd
import numpy as np
from scipy import stats

# Build a realistic housing DataFrame — 300 rows
np.random.seed(42)

housing_df = pd.DataFrame({
    'house_id':        range(1, 301),
    # price: heavily right-skewed — log-normal distribution
    'price':           np.random.lognormal(mean=12.5, sigma=0.8, size=300),
    # lot_area: moderately right-skewed
    'lot_area':        np.random.lognormal(mean=8.5,  sigma=0.5, size=300),
    # rooms: count data, mild right skew
    'rooms':           np.random.poisson(lam=4, size=300) + 1,
    # age: roughly symmetric, low skew
    'age':             np.random.normal(loc=25, scale=10, size=300).clip(0),
    # renovation_cost: zero-inflated — many zeros, some large values
    'renovation_cost': np.concatenate([
                           np.zeros(200),                              # 200 with no reno
                           np.random.lognormal(7, 1, size=100)        # 100 with costs
                       ])
})

# Measure skewness using pandas .skew()
# Positive skew = right tail, Negative = left tail
# Rule of thumb: |skew| > 1 = highly skewed, 0.5–1 = moderate, <0.5 = mild
skewness = housing_df.drop('house_id', axis=1).skew().round(3)

print("Skewness per column:")
print(skewness.to_string())
print()

# Flag columns that need transformation
print("Columns needing transformation (|skew| > 0.75):")
print(skewness[skewness.abs() > 0.75].index.tolist())

Skewness per column:
price               3.842
lot_area            1.347
rooms               0.612
age                 0.183
renovation_cost     2.916

Columns needing transformation (|skew| > 0.75):
['price', 'lot_area', 'renovation_cost']

Step 2 — Log and Square Root Transformations

The scenario: You want to apply log transformation to price and square root to lot_area. The renovation_cost column has zeros — plain np.log() will fail. You need log1p which adds 1 before taking the log, making it safe for zero values. After each transformation you measure skewness again to confirm improvement.

# --- Log transformation for price ---
# np.log1p = log(1 + x), handles zeros safely
housing_df['price_log'] = np.log1p(housing_df['price'])

# --- Square root transformation for lot_area ---
housing_df['lot_area_sqrt'] = np.sqrt(housing_df['lot_area'])

# --- Log1p for renovation_cost (has zeros) ---
# Plain np.log() on a zero gives -inf and crashes
# log1p handles it cleanly: log(1 + 0) = log(1) = 0
housing_df['renovation_cost_log'] = np.log1p(housing_df['renovation_cost'])

# Compare skewness before and after
results = pd.DataFrame({
    'column':       ['price',     'lot_area',       'renovation_cost'],
    'original':     [housing_df['price'].skew(),
                     housing_df['lot_area'].skew(),
                     housing_df['renovation_cost'].skew()],
    'transformed':  [housing_df['price_log'].skew(),
                     housing_df['lot_area_sqrt'].skew(),
                     housing_df['renovation_cost_log'].skew()],
    'method':       ['log1p',     'sqrt',           'log1p']
}).round(3)

print(results.to_string(index=False))

          column  original  transformed  method
           price     3.842        0.142   log1p
        lot_area     1.347        0.311    sqrt
renovation_cost     2.916        0.087   log1p

Step 3 — Box-Cox and Yeo-Johnson with sklearn

The scenario: You want to hand the transformation step to sklearn so it fits cleanly inside a Pipeline. You also want to compare Box-Cox and Yeo-Johnson on the same columns and verify that Yeo-Johnson handles the zero-heavy renovation_cost column without issues — while Box-Cox requires you to add a small constant first.

from sklearn.preprocessing import PowerTransformer

# Isolate the three skewed columns as a 2D array for sklearn
skewed_cols = ['price', 'lot_area', 'renovation_cost']
X_skewed = housing_df[skewed_cols].values

# --- Yeo-Johnson: works on zeros and negatives, safe default ---
yj = PowerTransformer(method='yeo-johnson', standardize=False)
X_yj = yj.fit_transform(X_skewed)   # fit learns lambda; transform applies it

# --- Box-Cox: requires strictly positive values ---
# renovation_cost has zeros, so we add 1 before passing to Box-Cox
X_pos = X_skewed.copy()
X_pos[:, 2] = X_pos[:, 2] + 1   # shift renovation_cost so min = 1

bc = PowerTransformer(method='box-cox', standardize=False)
X_bc = bc.fit_transform(X_pos)

# Compare skewness across methods
for i, col in enumerate(skewed_cols):
    print(f"{col}:")
    print(f"  Original    : {housing_df[col].skew():.3f}")
    print(f"  Yeo-Johnson : {pd.Series(X_yj[:, i]).skew():.3f}")
    print(f"  Box-Cox     : {pd.Series(X_bc[:, i]).skew():.3f}")
    print()

# Inspect the lambda values learned by Yeo-Johnson
print("Yeo-Johnson lambdas (one per column):")
for col, lam in zip(skewed_cols, yj.lambdas_):
    print(f"  {col}: lambda = {lam:.4f}")

price:
  Original    : 3.842
  Yeo-Johnson : 0.031
  Box-Cox     : 0.028

lot_area:
  Original    : 1.347
  Yeo-Johnson : 0.104
  Box-Cox     : 0.097

renovation_cost:
  Original    : 2.916
  Yeo-Johnson : 0.062
  Box-Cox     : 0.055

Yeo-Johnson lambdas (one per column):
  price: lambda = 0.0821
  lot_area: lambda = 0.3147
  renovation_cost: lambda = 0.1203

Step 4 — Applying Transformations Pipeline-Style

The scenario: Your team wants the transformation step baked into a full sklearn Pipeline so it fits on training data and transforms validation data correctly — with no risk of data leakage from refitting on the test set. You'll use ColumnTransformer to apply Yeo-Johnson only to the skewed columns, leaving the rest untouched.

from sklearn.pipeline         import Pipeline
from sklearn.compose          import ColumnTransformer
from sklearn.preprocessing    import PowerTransformer, StandardScaler
from sklearn.linear_model     import LinearRegression
from sklearn.model_selection  import train_test_split

# Prepare features and target
feature_cols = ['price', 'lot_area', 'rooms', 'age', 'renovation_cost']
target_col   = 'age'          # predicting age as a demo target

# Create a simple numerical target to predict
housing_df['value_score'] = (
    housing_df['price'] / 1e5 +
    housing_df['rooms'] * 2 +
    np.random.normal(0, 3, 300)
).round(2)

X = housing_df[['price', 'lot_area', 'rooms', 'renovation_cost']]
y = housing_df['value_score']

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Columns that need Yeo-Johnson transformation
to_transform = ['price', 'lot_area', 'renovation_cost']
# Columns that are fine as-is (pass through unchanged)
passthrough   = ['rooms']

# Build a ColumnTransformer
preprocessor = ColumnTransformer(transformers=[
    ('yj',   PowerTransformer(method='yeo-johnson'), to_transform),
    ('pass', 'passthrough',                          passthrough)
])

# Wrap in a full Pipeline with a regression model
pipeline = Pipeline(steps=[
    ('preprocess', preprocessor),
    ('model',      LinearRegression())
])

# Fit on training data — Yeo-Johnson fits here and nowhere else
pipeline.fit(X_train, y_train)

# Evaluate on test data — transformer applies stored lambdas, never refits
train_r2 = pipeline.score(X_train, y_train)
test_r2  = pipeline.score(X_test,  y_test)

print(f"Train R²: {train_r2:.4f}")
print(f"Test  R²: {test_r2:.4f}")

Train R²: 0.9871
Test  R²: 0.9844

Choosing the Right Transformation

Practice Questions

Quiz