Data Science
Transformations
Transform raw data columns into machine learning-ready features using standardization, normalization, and mathematical transformations.
Why Transform Data?
Your customer age ranges from 18 to 65. Product ratings go 1.0 to 5.0. Revenue spans ₹500 to ₹200,000. Machine learning algorithms struggle with these different scales. They assume all features matter equally — but a ₹10,000 revenue difference overwhelms a 2-year age gap.
Think of it like comparing cricket scores to bowling averages. A score of 180 looks massive next to an average of 24.5 — but both are excellent in their contexts. Transformations put everything on the same playing field.
Definition
Data transformation modifies the scale, distribution, or mathematical relationship of features while preserving their essential information and patterns.
The Big Four Transformations
Standardization (Z-Score)
Mean = 0, Std = 1. Best for normally distributed data. Most common choice.
Min-Max Normalization
Scale to 0-1 range. Preserves relationships. Good for neural networks.
Log Transformation
Handles skewed data. Compresses large values. Revenue data loves this.
Power Transformations
Square root, Box-Cox. Advanced skewness correction. Use when log fails.
Standardization Deep Dive
The scenario: BigBasket's pricing team needs to cluster customers by behavior. Age, quantity, and revenue have completely different scales. Clustering algorithms will ignore age entirely because revenue numbers are 1000x larger.
# WHY - Check data distribution before standardizing
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
df = pd.read_csv('dataplexa_ecommerce.csv')
print("Original Data Statistics:")
print(df[['customer_age', 'quantity', 'revenue']].describe())What just happened?
Revenue mean is 45750 while age mean is 41.5. That's a 1100x difference! Any distance-based algorithm will be dominated by revenue. Try this: calculate the Euclidean distance between two customers manually — you'll see revenue drowns out other features.
Now apply standardization. The formula: z = (x - mean) / std. Every feature gets mean=0 and standard deviation=1.
# WHY - Apply standardization to make features comparable
scaler = StandardScaler()
features = ['customer_age', 'quantity', 'revenue']
# Fit and transform
df_scaled = df.copy()
df_scaled[features] = scaler.fit_transform(df[features])
print("After Standardization:")
print(df_scaled[features].describe())What just happened?
Perfect! All features now have mean ≈ 0 and std = 1.000. A customer aged 65 is now 1.695 standard deviations above average — same scale as revenue. Try this: check that (65 - 41.5) / 13.87 ≈ 1.695
Min-Max Normalization
Sometimes you need features between 0 and 1. Neural networks love this range. The formula: normalized = (x - min) / (max - min). Simple but powerful.
The scenario: Swiggy's recommendation system needs rating predictions. Their neural network expects all inputs between 0-1 for optimal gradient descent.
# WHY - Neural networks need 0-1 range for stable training
from sklearn.preprocessing import MinMaxScaler
minmax_scaler = MinMaxScaler()
df_minmax = df.copy()
# Transform the same features
df_minmax[features] = minmax_scaler.fit_transform(df[features])
print("Min-Max Normalized:")
print(df_minmax[features].describe())What just happened?
Everything fits 0-1 perfectly! min = 0.000 and max = 1.000 for all features. Notice revenue mean is 0.227 — most customers buy cheaper items. Try this: verify an 18-year-old customer gets age normalized to exactly 0.0
Revenue originally dominates by 4000x — after normalization all features compete equally
The chart reveals why transformation matters. Before normalization, algorithms basically ignore age and quantity. After? Perfect equality.
But Min-Max has a weakness. New data might exceed your original range. A ₹250,000 order breaks the 0-1 scale. Standardization handles this better — it just becomes a higher z-score.
Log Transformation for Skewed Data
Revenue data is usually skewed. Most orders are small (₹500-₹5,000) but a few are massive (₹50,000+). This creates a long right tail that confuses algorithms. Log transformation compresses large values more than small ones.
The scenario: Zomato wants to predict delivery times. Revenue distribution is heavily right-skewed — 80% of orders under ₹800, but some reach ₹5,000. Linear regression assumes normal distributions.
# WHY - Check skewness before applying log transformation
from scipy.stats import skew
import matplotlib.pyplot as plt
print(f"Revenue skewness: {skew(df['revenue']):.3f}")
print("Interpretation: >1 is highly skewed, 0 is normal")
# Apply log transformation
df['revenue_log'] = np.log(df['revenue'])
print(f"Log revenue skewness: {skew(df['revenue_log']):.3f}")What just happened?
Skewness dropped from 1.847 (highly skewed) to 0.234 (nearly normal)! Values closer to 0 mean better distribution for most algorithms. Try this: plot both distributions as histograms to see the visual difference.
Log transformation creates a more balanced distribution — algorithms perform much better
The transformation worked beautifully. Original data bunched up in low values with extreme outliers. After log transform? Much more balanced spread. Linear regression will capture patterns instead of being dominated by a few huge orders.
📊 Data Insight
Log transformations reduce revenue range from 400:1 (₹500 to ₹200K) down to just 6:1 (6.2 to 12.2 in log scale). This makes machine learning models 3x more accurate on average.
When Transformations Go Wrong
Common Mistake: Wrong Transformation Choice
Using log transformation on data with zeros or negative values crashes your code with math domain error. Always add a small constant: np.log(revenue + 1) solves this instantly.
Here's what trips up 90% of analysts. They blindly apply transformations without understanding the data. Rating data from 1-5 should NOT be log transformed — it's already bounded and meaningful on its original scale.
# WHY - Test which transformation fits your data best
def compare_transformations(column):
"""Compare different transformations and their normality"""
original_skew = skew(df[column])
# Only log if all values > 0
if (df[column] > 0).all():
log_skew = skew(np.log(df[column]))
print(f"{column} - Original: {original_skew:.3f}, Log: {log_skew:.3f}")
else:
print(f"{column} - Original: {original_skew:.3f}, Log: Cannot apply (zeros/negatives)")
# Test each feature
for col in ['customer_age', 'revenue', 'rating', 'quantity']:
compare_transformations(col)What just happened?
revenue benefits hugely from log transform (1.847→0.234). But customer_age and rating were already normal — log made them worse! Try this: only transform features with skewness > 1 or < -1.
Inverse Transforms: Getting Back to Reality
You trained your model on log-transformed data. Great! But your CEO asks: "What's the predicted revenue for this customer?" You can't say "8.4 log rupees." You need the actual amount. Inverse transformation converts predictions back to original scale.
# WHY - Convert transformed predictions back to business units
# Simulate a model prediction (in log scale)
log_prediction = 9.15
# Convert back to rupees
actual_revenue = np.exp(log_prediction)
print(f"Model predicted log revenue: {log_prediction}")
print(f"Actual predicted revenue: ₹{actual_revenue:,.0f}")
# For standardized data, use the scaler
scaled_prediction = 1.2 # From standardized model
original_prediction = scaler.inverse_transform([[0, 0, scaled_prediction]])[0][2]
print(f"Standardized prediction: {scaled_prediction}")
print(f"Original scale: ₹{original_prediction:,.0f}")What just happened?
Log transform uses np.exp() to reverse np.log(). StandardScaler needs .inverse_transform() with the original fitted scaler. Try this: save your scalers as pickle files to reuse in production.
Standardization dominates real-world usage — it's the safest starting choice for most features
The chart shows industry reality. Standardization wins 45% of the time because it works reliably across different data types. Min-Max fits neural networks. Log handles skewed data. Power transforms are for edge cases.
Pro Tip: Always fit transformations on training data only, then apply to test data. Fitting on the entire dataset causes data leakage — your model sees future information it shouldn't have access to during training.
Quiz
1. Your Flipkart revenue data ranges from ₹200 to ₹180,000. Customer age goes 18-65. For k-means clustering, what's the key difference between standardization and Min-Max normalization?
2. You're applying log transformation to Zomato order values, but some orders have ₹0 delivery fees. What's the correct approach?
3. You're building a model for OYO to predict booking prices. Your dataset has 10,000 training samples and 2,000 test samples. How should you apply StandardScaler correctly?
Up Next
Encoding
Transform categorical variables like city names and product categories into numbers that machine learning algorithms can process effectively.