ML Algorithms Cheat Sheet — Regression · Trees · SVM · KNN · Naive Bayes · Ensembles | Dataplexa

ML Algorithms Cheat Sheet

Linear Regression · Logistic Regression · Decision Trees · SVM · KNN · Naive Bayes · Ensembles

Sheet 2 of 4 Machine Learning Intermediate Printable

Algorithm Quick-Selector

When to Use What

Algorithm	Task	Data Size	Interpretable?	Key Strength	Key Weakness
Linear Regression	Regression	Any	✅ Yes	Fast, simple baseline	Assumes linearity
Logistic Regression	Classification	Any	✅ Yes	Probability output, fast	Linear boundary only
Decision Tree	Both	Small–Med	✅ Yes	No scaling needed, visual	Overfits easily
Random Forest	Both	Med–Large	⚠ Partial	Robust, low variance	Slow on large data
Gradient Boosting	Both	Med–Large	⚠ Partial	High accuracy, tabular SOTA	Many hyperparams, slow train
SVM	Classification	Small–Med	❌ No	Effective in high dims	Slow on large n, kernel choice
KNN	Both	Small	✅ Yes	No training phase	Slow at prediction, sensitive to scale
Naive Bayes	Classification	Any	✅ Yes	Very fast, NLP-friendly	Feature independence assumption

Rule of thumb: Always start with Logistic Regression (classification) or Linear Regression (regression) as a baseline before moving to complex models.

Linear Regression

Regression

Model Equation

ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ

Fit by minimizing MSE · β found via OLS or gradient descent

OLS Closed-Form Solution

β = (XᵀX)⁻¹ Xᵀy

Works when n > p and XᵀX is invertible · O(p³) — use GD for large p

Assumption	Check With
Linearity	Residual vs fitted plot
Independence of errors	Durbin-Watson test
Homoscedasticity	Scale-location plot
Normality of residuals	Q-Q plot, Shapiro-Wilk
No multicollinearity	VIF < 10

sklearn

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
model.coef_       # β₁…βₙ
model.intercept_  # β₀

Logistic Regression

Classification

Sigmoid Output

p = σ(z) = 1 / (1 + e⁻ᶻ) where z = Xβ

Output is P(y=1|X) · Predict class 1 if p ≥ 0.5 (adjustable threshold)

Log-Odds (Logit)

log[p / (1−p)] = β₀ + β₁x₁ + … + βₙxₙ

eᵝ = odds ratio — how much odds multiply per unit increase in x

Variant	Use When
Binary LR	2 classes (sigmoid output)
Multinomial LR	3+ classes — softmax output
Ordinal LR	Ordered categories

sklearn

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(C=1.0,  # C = 1/λ
  solver='lbfgs', max_iter=1000)
model.predict_proba(X_test)  # probabilities

Decision Trees

Both Tasks

Gini Impurity (Classification)

Gini = 1 − Σ pᵢ²

Split on feature that maximises information gain = parent impurity − weighted child impurity

Entropy / Information Gain

H = −Σ pᵢ log₂(pᵢ)

IG = H(parent) − [weighted avg H(children)]

Hyperparameter	Effect
max_depth	Limits tree depth — prevents overfitting
min_samples_split	Min samples to split a node
min_samples_leaf	Min samples in a leaf node
criterion	gini or entropy (classification); mse (regression)

✅ Pros

No scaling needed
Handles mixed types
Highly interpretable

❌ Cons

High variance (overfits)
Unstable — small data changes = big tree changes

Random Forest

Ensemble

Bagging (Bootstrap Aggregating) of many decision trees + random feature subsets at each split.

Prediction

ŷ = majority_vote(T₁(x), T₂(x), …, Tₙ(x))

Classification: majority vote · Regression: mean of all tree predictions

Hyperparameter	Typical Value	Effect
n_estimators	100–500	More trees = lower variance (diminishing returns)
max_features	√p (clf) · p/3 (reg)	Controls diversity between trees
max_depth	None (default)	Deeper trees = lower bias
oob_score	True	Free validation using out-of-bag samples

sklearn

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
  n_estimators=200, max_features='sqrt',
  oob_score=True, n_jobs=-1)
rf.feature_importances_  # Gini importance

Gradient Boosting & XGBoost

Ensemble

Sequentially fits new trees to the residuals of previous trees — boosting = additive model.

Additive Update

F_m(x) = F_{m-1}(x) + α · hₘ(x)

α = learning rate · hₘ = new tree fitted to negative gradient of loss

Hyperparameter	Typical Range	Controls
n_estimators	100–1000	Number of boosting rounds
learning_rate	0.01–0.3	Shrinkage — lower = more robust
max_depth	3–6	Tree complexity (shallower = better)
subsample	0.6–0.9	Row sampling per tree — reduces variance
colsample_bytree	0.6–0.9	Feature sampling per tree
reg_alpha / lambda	0–1	L1 / L2 regularization on leaf weights

XGBoost

import xgboost as xgb
model = xgb.XGBClassifier(
  n_estimators=300, learning_rate=0.05,
  max_depth=4, subsample=0.8,
  eval_metric='logloss',
  early_stopping_rounds=20)

LightGBM vs XGBoost: LightGBM is faster on large datasets (leaf-wise growth). XGBoost is more tuned for accuracy. Both beat vanilla GBM.

Support Vector Machine (SVM)

Classification

Max-Margin Objective

maximize 2/‖w‖ subject to yᵢ(w·xᵢ + b) ≥ 1

Decision boundary: w·x + b = 0 · Support vectors = closest points to boundary

Soft Margin (C parameter)

min ½‖w‖² + C·Σξᵢ

High C = hard margin (low bias, high variance) · Low C = wide margin (high bias, low variance)

Kernel	Formula	Use When
Linear	xᵀz	Linearly separable, high-dim (text)
RBF / Gaussian	exp(−γ‖x−z‖²)	Default — non-linear, general
Polynomial	(γxᵀz + r)ᵈ	Image classification
Sigmoid	tanh(γxᵀz + r)	Neural net approximation

Always scale features before SVM — RBF kernel is distance-based and sensitive to feature magnitude.

K-Nearest Neighbours (KNN)

Both Tasks

No training phase — prediction looks up the k closest training points and aggregates their labels.

Euclidean Distance (default)

d(x, z) = √[Σᵢ (xᵢ − zᵢ)²]

Also: Manhattan (L1) · Minkowski (generalises both) · Cosine for text

k value	Effect
k = 1	Low bias, very high variance — noisy decision boundary
Small k	Complex boundary — overfits
Large k	Smooth boundary — may underfit; slow prediction
k = √n	Common rule of thumb for starting point

sklearn

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(
  n_neighbors=5, metric='euclidean',
  weights='distance')  # closer = more weight

Scale features! KNN is fully distance-based — unscaled features with large ranges dominate the distance.

Naive Bayes

Classification

Bayes' Theorem

P(y|X) ∝ P(y) · Π P(xᵢ|y)

"Naive" = assumes features are conditionally independent given class

Variant	Likelihood P(xᵢ\|y)	Best For
Gaussian NB	Normal distribution	Continuous features
Multinomial NB	Multinomial distribution	Word counts, text classification
Bernoulli NB	Bernoulli distribution	Binary features (word presence)

sklearn — Text Classification

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline

pipe = Pipeline([
  ('tfidf', TfidfVectorizer()),
  ('clf',  MultinomialNB(alpha=1.0))
])

Laplace smoothing (alpha=1): Prevents zero probability for unseen words in test data.

Ensemble Methods Compared

Ensemble

Method	Strategy	Reduces	Trees Built
Bagging	Parallel trees on bootstrap samples	Variance	Independently
Random Forest	Bagging + random feature subsets	Variance + correlation	Independently
Boosting	Sequential — each tree fixes residuals	Bias	Sequentially
Stacking	Meta-learner on base model predictions	Both	Independently
Voting	Majority vote / average of diverse models	Variance	Independently

Voting Classifier — sklearn

from sklearn.ensemble import VotingClassifier
vc = VotingClassifier(estimators=[
  ('lr',  LogisticRegression()),
  ('rf',  RandomForestClassifier()),
  ('xgb', XGBClassifier())],
  voting='soft')  # use probabilities

Tabular data winner: Gradient Boosting (XGBoost / LightGBM / CatBoost) consistently wins on structured tabular data competitions.

Regularised Linear Models

Regression

Model	Penalty	Effect on Coefficients
Ridge (L2)	λΣβᵢ²	Shrinks all — never exactly zero
Lasso (L1)	λΣ\|βᵢ\|	Pushes some to exactly zero — feature selection
ElasticNet	α·L1 + (1−α)·L2	Grouped selection — handles correlated features

sklearn

from sklearn.linear_model import Ridge, Lasso, ElasticNet

Ridge(alpha=1.0)         # alpha = λ
Lasso(alpha=0.1)         # sparse coefficients
ElasticNet(alpha=0.5,
  l1_ratio=0.5)          # 50% L1, 50% L2

# Auto-tune alpha with CV
from sklearn.linear_model import RidgeCV
RidgeCV(alphas=[0.1, 1.0, 10.0])

Use Lasso when you suspect only a few features matter. Use Ridge when all features likely contribute. ElasticNet when features are correlated.

Complexity & Scalability

Reference

Algorithm	Train Time	Predict Time	Memory
Linear / Logistic Reg	O(np)	O(p)	Low
Decision Tree	O(np log n)	O(depth)	Low
Random Forest	O(T·np log n)	O(T·depth)	High
Gradient Boosting	O(T·np)	O(T·depth)	Medium
SVM (RBF)	O(n²–n³)	O(n_sv · p)	Medium
KNN	O(1)	O(n·p)	High (stores all)
Naive Bayes	O(np)	O(p)	Low

n = samples · p = features · T = number of trees · n_sv = support vectors

SVM bottleneck: Becomes impractical above ~50K samples with RBF kernel — use LinearSVC or kernel approximations instead.

ML Algorithms Mastery Checklist

Self-Assessment

Linear Models

Derive the OLS solution β = (XᵀX)⁻¹Xᵀy and state its assumptions

Explain the sigmoid function and why it maps any value to (0, 1)

Interpret logistic regression coefficients as log-odds and odds ratios

Compare Ridge, Lasso, and ElasticNet and choose appropriately

Check linear regression assumptions with diagnostic plots

Trees & Ensembles

Explain Gini impurity and Information Gain for tree splits

Describe how Random Forest reduces variance through bagging

Explain the boosting residual-fitting mechanism

Tune XGBoost hyperparameters (learning_rate, max_depth, subsample)

Distinguish bagging vs boosting vs stacking

SVM & KNN

Explain the max-margin objective and role of support vectors

Describe the kernel trick and when to use RBF vs linear kernel

Tune C and γ for SVM via grid search

Explain KNN prediction and why scaling is mandatory

Choose k using cross-validation; understand the bias-variance effect

Naive Bayes

Apply Bayes' theorem to derive the classification rule

State the naive independence assumption and its practical implications

Choose the correct NB variant (Gaussian / Multinomial / Bernoulli)

Explain Laplace smoothing and why it's needed

Selection & Implementation

Select the right algorithm given task, data size, and interpretability needs

Set up a full sklearn pipeline (scaler + model) to prevent leakage

Implement cross-validated hyperparameter search for any algorithm

Extract and interpret feature importances from tree-based models

Know when to use predict() vs predict_proba()

Complexity & Tradeoffs

Recall training and prediction time complexity for each algorithm

Identify when SVM becomes impractical and what to use instead

Explain why KNN has no training cost but high prediction cost

Build and evaluate a voting / stacking ensemble

▶ Next Up: Sheet 3 — Model Evaluation

Accuracy · Precision · Recall · F1 · AUC-ROC · RMSE · Confusion Matrix · Cross-Validation

Model Evaluation →

← Back