Feature Engineering Cheat Sheet — Encoding · Scaling · Imputation · Selection · PCA | Dataplexa

Feature Engineering Cheat Sheet

Encoding · Scaling · Imputation · Feature Creation · Selection · Dimensionality Reduction · Pipelines

Sheet 4 of 4 Machine Learning Intermediate Printable

Feature Engineering — Overview

Foundation

Core Idea

Better features → better model, always

A simple model with great features beats a complex model with raw data

Step	Task
1. Understand	Explore data — distributions, types, missing, correlations
2. Clean	Handle missing values, outliers, duplicates, wrong types
3. Encode	Convert categoricals to numbers the model understands
4. Scale	Normalise / standardise numerical features
5. Create	Derive new features from existing ones
6. Select	Remove irrelevant / redundant features
7. Reduce	Compress feature space (PCA, t-SNE)

Feature Type	Examples	Actions
Numerical	age, price, count	Scale, bin, log-transform, clip
Categorical	city, color, brand	One-hot, ordinal, target encode
Ordinal	rating, education level	Ordinal encode, keep order
Datetime	timestamp, date	Extract hour, day, month, week, cyclic encode
Text	reviews, descriptions	TF-IDF, word embeddings, n-grams
Boolean	is_active, has_promo	Keep as 0/1 directly
High-cardinality	ZIP code, user_id	Target encode, frequency encode, hash

Algorithm	Needs Scaling?	Needs Encoding?
Linear / Logistic Reg	✅ Yes	✅ Yes
SVM	✅ Yes	✅ Yes
KNN	✅ Yes	✅ Yes
Neural Networks	✅ Yes	✅ Yes
Decision Tree	❌ No	✅ Yes
Random Forest	❌ No	✅ Yes
XGBoost / LGBM	❌ No	⚠ Often built-in
Naive Bayes	❌ No	✅ Yes

Rule: Distance-based and gradient-based algorithms need scaling. Tree-based algorithms generally don't.

Encoding Categorical Variables

Encoding

Method	When to Use	Creates
One-Hot	Nominal, low cardinality (<20 cats)	Binary columns per category
Ordinal	Ordered categories (low/mid/high)	Single integer column
Label	Tree models only — no order implied	Single integer column
Target	High cardinality, regression/classification	Mean target per category
Frequency	High cardinality, quick baseline	Category count/frequency
Binary	Very high cardinality, memory-efficient	Binary bit columns
Hashing	Extremely high cardinality, NLP	Fixed-size hash vector

sklearn Encoding

from sklearn.preprocessing import (
  OneHotEncoder, OrdinalEncoder, LabelEncoder)

# One-Hot — drop first to avoid dummy trap
OneHotEncoder(drop='first', sparse_output=False)

# Ordinal — specify order explicitly
OrdinalEncoder(categories=[['low','mid','high']])

# Target encoding (category_encoders)
from category_encoders import TargetEncoder
TargetEncoder(smoothing=1.0)

Dummy trap: Always drop one column with One-Hot Encoding for linear models — use drop='first'. Tree models don't need this.

Feature Scaling

Scaling

Method	Formula	Output Range	Use When
Min-Max	(x−min)/(max−min)	[0, 1]	Bounded output needed, no outliers
Standardization	(x−μ)/σ	~(−3, 3)	Normal distribution, outliers ok
Robust Scaler	(x−median)/IQR	Unbounded	Heavy outliers present
Max-Abs Scaler	x / max(\|x\|)	[−1, 1]	Sparse data — doesn't center
Log Transform	log(x + 1)	Unbounded	Right-skewed (income, counts)
Box-Cox	Optimal power transform	Unbounded	Positive values, normalise distribution

sklearn

from sklearn.preprocessing import (
  StandardScaler, MinMaxScaler,
  RobustScaler, PowerTransformer)

StandardScaler()          # z-score
MinMaxScaler(feature_range=(0,1))
RobustScaler()            # IQR-based
PowerTransformer(method='box-cox')

Critical: Always fit on training data only, then transform both train and test. Never fit on test — it causes data leakage.

Missing Value Imputation

Imputation

Strategy	Fill With	Best For
Mean	Column mean	Numerical, normally distributed, few missing
Median	Column median	Numerical with outliers or skewed
Mode	Most frequent value	Categorical features
Constant	Fixed value (0, "Unknown")	When missing = meaningful signal
KNN Imputer	k-nearest neighbours	Small datasets, preserves relationships
Iterative	Model each feature	Complex MAR data — best but slow

sklearn

from sklearn.impute import (
  SimpleImputer, KNNImputer, IterativeImputer)

SimpleImputer(strategy='median')
SimpleImputer(strategy='most_frequent')
SimpleImputer(strategy='constant',
  fill_value='Unknown')
KNNImputer(n_neighbors=5)
IterativeImputer(max_iter=10)

Missing indicator: Add a binary column flagging which rows were imputed — the missingness itself may be predictive.

Feature Creation

Engineering

Technique	Example
Arithmetic	price_per_sqft = price / area
Ratios	ctr = clicks / impressions
Binning	age → [0–18, 19–35, 36–60, 60+]
Log transform	log1p(revenue) — normalise skew
Polynomial	x², x·y — capture non-linearity
Interaction	age × income — joint effect
Datetime	hour, dayofweek, month, is_weekend, quarter
Cyclic encode	sin/cos of hour, month — preserves circularity
Aggregations	user mean spend, category purchase count
Lag features	previous day's sales, rolling 7-day avg

Cyclic Encoding — Hours

import numpy as np
# Hour 0 and 23 are close — sin/cos keeps this
df['hour_sin'] = np.sin(2*np.pi*df['hour']/24)
df['hour_cos'] = np.cos(2*np.pi*df['hour']/24)

Handling Outliers

Data Quality

Detection Method	Rule	Best For
IQR Method	Outside [Q1−1.5·IQR, Q3+1.5·IQR]	Skewed distributions
Z-Score	\|z\| > 3	Normally distributed data
Modified Z-Score	\|0.6745·(x−median)/MAD\| > 3.5	Small datasets, robust
Isolation Forest	ML-based anomaly detection	High-dimensional data

Treatment	When
Remove	Clearly wrong / data entry error
Cap / Winsorise	Keep value, clip to percentile (e.g. 1st–99th)
Log transform	Compress scale without removing
Robust model	Use MAE loss or tree-based models — outlier-resistant
Keep + flag	Add binary is_outlier feature — signal may be useful

Winsorising with pandas

lower = df['price'].quantile(0.01)
upper = df['price'].quantile(0.99)
df['price'] = df['price'].clip(lower, upper)

Feature Selection

Selection

Method	Type	How
Variance Threshold	Filter	Remove near-zero variance features
Correlation Filter	Filter	Remove features correlated > 0.9 with another
Chi-Square	Filter	Test independence with target (classification)
ANOVA F-test	Filter	Test mean differences across classes
Mutual Information	Filter	Captures non-linear relationships
RFE	Wrapper	Recursively removes weakest features
Lasso (L1)	Embedded	Shrinks irrelevant coefficients to zero
Tree Importance	Embedded	Feature importance from RF / XGBoost
SHAP	Embedded	Model-agnostic importance — most reliable

sklearn — SelectKBest + RFE

from sklearn.feature_selection import (
  SelectKBest, f_classif,
  RFE, mutual_info_classif)

# Filter — top 10 by ANOVA F
SelectKBest(f_classif, k=10)

# Wrapper — recursive elimination
RFE(estimator=LogisticRegression(),
  n_features_to_select=10)

Dimensionality Reduction

Reduction

Method	Type	Best For
PCA	Linear	Variance preservation, multicollinearity, preprocessing
t-SNE	Non-linear	Visualisation (2D/3D) — not for preprocessing
UMAP	Non-linear	Faster t-SNE, preserves global structure better
LDA	Supervised	Maximise class separability
Autoencoder	Neural	Complex non-linear compression
Truncated SVD	Linear	Sparse data (NLP/TF-IDF)

PCA — Explained Variance

Choose n_components where Σ explained_variance_ratio_ ≥ 0.95

Each principal component is orthogonal — no multicollinearity

sklearn PCA

from sklearn.decomposition import PCA

# Keep 95% variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X_train)

# Explained variance per component
pca.explained_variance_ratio_
pca.n_components_  # how many chosen

PCA requires scaling first — features must be standardised or PCA will be dominated by high-magnitude features.

Full sklearn Pipeline — Numeric + Categorical

Pipeline

Complete Production Pipeline

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import (
  StandardScaler, OneHotEncoder)
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

num_cols = ['age', 'income', 'score']
cat_cols = ['city', 'gender']

# Numeric pipeline
num_pipe = Pipeline([
  ('impute', SimpleImputer(strategy='median')),
  ('scale',  StandardScaler())
])

# Categorical pipeline
cat_pipe = Pipeline([
  ('impute', SimpleImputer(strategy='most_frequent')),
  ('encode', OneHotEncoder(drop='first',
               handle_unknown='ignore'))
])

Assemble + Train + Evaluate

# Combine with ColumnTransformer
preprocessor = ColumnTransformer([
  ('num', num_pipe, num_cols),
  ('cat', cat_pipe, cat_cols)
])

# Full pipeline with model
pipe = Pipeline([
  ('prep',  preprocessor),
  ('model', RandomForestClassifier(
             n_estimators=200))
])

# Fit — all transforms fit on train only
pipe.fit(X_train, y_train)

# Evaluate — transform applied automatically
pipe.score(X_test, y_test)

# Cross-validate the full pipeline
from sklearn.model_selection import cross_val_score
cross_val_score(pipe, X, y, cv=5)

Why pipelines? Prevents data leakage — scalers and encoders fit only on training folds inside CV, not on the full dataset. Always use a pipeline in production.

Class Imbalance in Features

Imbalance

Technique	How	When
SMOTE	Synthetic minority oversampling	Tabular, severe imbalance
ADASYN	Adaptive synthetic sampling	Complex decision boundaries
Random Undersampling	Remove majority samples	Large dataset
Class Weights	class_weight='balanced'	Quick, no data modification
Threshold Tuning	Lower decision threshold	Post-training adjustment

imbalanced-learn — SMOTE Pipeline

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE

pipe = ImbPipeline([
  ('prep',  preprocessor),
  ('smote', SMOTE(random_state=42)),
  ('model', RandomForestClassifier())
])

Use imblearn Pipeline not sklearn Pipeline when SMOTE is involved — it applies oversampling only to training folds, not validation folds inside CV.

Text & Datetime Features

Special Types

Datetime Feature Extraction

df['year']       = df['date'].dt.year
df['month']      = df['date'].dt.month
df['dayofweek']  = df['date'].dt.dayofweek
df['is_weekend'] = df['dayofweek'] >= 5
df['quarter']    = df['date'].dt.quarter
df['days_since']  = (pd.Timestamp.now()
                   - df['date']).dt.days

Text — TF-IDF + Bag of Words

from sklearn.feature_extraction.text import (
  TfidfVectorizer, CountVectorizer)

# Bag of Words
CountVectorizer(max_features=5000,
  ngram_range=(1,2))  # unigrams+bigrams

# TF-IDF (preferred)
TfidfVectorizer(max_features=5000,
  stop_words='english',
  sublinear_tf=True)

Word Embeddings: For richer text features use pre-trained embeddings — Word2Vec, GloVe, FastText, or sentence-transformers (BERT-based).

Feature Engineering Mastery Checklist

Self-Assessment

Encoding & Scaling

Choose the right encoding for nominal, ordinal, and high-cardinality features

Apply One-Hot encoding and drop the first column to avoid the dummy trap

Use Target Encoding with smoothing to avoid target leakage

Apply StandardScaler, MinMaxScaler, and RobustScaler correctly

Know which algorithms need scaling and which don't

Apply log/Box-Cox transform to right-skewed features

Imputation

Impute numerical features with median (outlier-robust)

Impute categorical features with mode or constant

Add a missing indicator flag when missingness is informative

Use KNNImputer or IterativeImputer for complex missing patterns

Feature Creation

Create ratio and interaction features from domain knowledge

Extract datetime components (hour, dayofweek, is_weekend, quarter)

Apply cyclic encoding (sin/cos) for periodic features like hour and month

Bin continuous features when relationship with target is non-monotonic

Create lag and rolling window features for time-series data

Outliers

Detect outliers using IQR and Z-score methods

Winsorise features at 1st and 99th percentile

Decide between removing, capping, transforming, or flagging outliers

Selection & Reduction

Remove near-zero variance and highly correlated features

Apply SelectKBest with f_classif or mutual_info_classif

Use RFE to recursively select the most important features

Use Lasso or tree importance for embedded feature selection

Apply PCA after scaling and choose n_components for 95% variance

Use SHAP values for reliable model-agnostic feature importance

Pipelines

Build a ColumnTransformer for separate numeric and categorical pipelines

Wrap everything in a sklearn Pipeline to prevent leakage in CV

Use imblearn Pipeline when SMOTE is part of the workflow

Cross-validate the full pipeline — not just the model step

🎉 You've completed the ML Fundamentals series!

Next: explore Deep Learning, NLP, Computer Vision, Time Series, and Reinforcement Learning series on Dataplexa.

Browse All Cheat Sheets →

← Back