ML Basics Cheat Sheet
Supervised Β· Unsupervised Β· Loss Functions Β· Bias-Variance Β· Regularization Β· Model Workflow
Sheet 1 of 4
Machine Learning
BeginnerβIntermediate
Printable
What is Machine Learning?
DefinitionCore Idea
Learn f: X β Y from data, without explicit rules
X = input features Β· Y = target output Β· f = learned function (model)
| Type | Labels? | Goal | Example |
|---|---|---|---|
| Supervised | β Yes | Predict Y from X | Spam detection, price prediction |
| Unsupervised | β No | Find structure in X | Customer segmentation, anomaly detection |
| Semi-supervised | Partial | Leverage few labels + many unlabeled | Image classification with sparse labels |
| Reinforcement | Rewards | Maximize cumulative reward | Game playing, robotics |
Key insight: ML finds patterns automatically β the model improves as it sees more data.
Supervised Learning
Coreπ· Classification
- Output: discrete class
- Binary: yes/no, spam/ham
- Multi-class: digit 0β9
- Metric: Accuracy, F1, AUC
- Algorithms: LR, SVM, Trees, KNN
π Regression
- Output: continuous value
- House price, temperature
- Stock forecast, demand
- Metric: MSE, RMSE, MAE, RΒ²
- Algorithms: Linear Reg, SVR, Trees
General Training Objective
minimize L(Ε·, y) over model parameters ΞΈ
Ε· = model prediction Β· y = true label Β· L = loss function
Tip: Always split data into train / validation / test before fitting β never evaluate on training data.
Unsupervised Learning
Core| Task | Goal | Common Algorithms |
|---|---|---|
| Clustering | Group similar data points | K-Means, DBSCAN, Hierarchical |
| Dimensionality Reduction | Compress features, remove noise | PCA, t-SNE, UMAP, Autoencoders |
| Density Estimation | Model data distribution | GMM, KDE, Normalizing Flows |
| Anomaly Detection | Identify outliers | Isolation Forest, One-Class SVM |
| Association Rules | Find co-occurrence patterns | Apriori, FP-Growth |
Use case: When labels are expensive or unavailable β explore structure in raw data first.
Loss Functions
MathLoss measures how far predictions are from the truth. Training minimizes it.
| Loss | Formula | Use When |
|---|---|---|
| MSE | 1/n Β· Ξ£(yα΅’ β Ε·α΅’)Β² | Regression, penalizes large errors heavily |
| MAE | 1/n Β· Ξ£|yα΅’ β Ε·α΅’| | Regression, robust to outliers |
| Huber | MSE if |e|β€Ξ΄, else MAE | Regression, balance of MSE+MAE |
| Binary Cross-Entropy | β[yΒ·log(Ε·) + (1βy)Β·log(1βΕ·)] | Binary classification |
| Categorical Cross-Entropy | βΞ£ yα΅’ Β· log(Ε·α΅’) | Multi-class classification |
| Hinge | max(0, 1 β yΒ·Ε·) | SVM classification |
Log Loss (Binary Cross-Entropy) Intuition
L = β(yΒ·log p + (1βy)Β·log(1βp))
p = predicted probability Β· heavily penalizes confident wrong predictions
BiasβVariance Tradeoff
TheoryError Decomposition
Total Error = BiasΒ² + Variance + Irreducible Noise
Irreducible noise = inherent randomness in data β cannot be removed
β¬ High Bias (Underfitting)
- Model too simple
- High train & test error
- Misses patterns in data
- Fix: add complexity, more features
β¬ High Variance (Overfitting)
- Model too complex
- Low train, high test error
- Memorizes training noise
- Fix: regularize, get more data
Sweet spot: Choose the simplest model that generalizes well to unseen data.
Overfitting vs Underfitting
Diagnosis| Signal | Underfitting | Overfitting |
|---|---|---|
| Train error | High β | Low β |
| Val/Test error | High β | High β |
| TrainβVal gap | Small | Large |
| Root cause | Too simple | Too complex |
| Fix Underfitting | Fix Overfitting |
|---|---|
| Increase model complexity | Add regularization (L1/L2) |
| Add more features | Reduce features / use selection |
| Train longer / more epochs | Early stopping |
| Remove strong regularization | Collect more training data |
| Try more powerful algorithm | Dropout (neural networks) |
Regularization
TechniqueRegularized Loss
L_reg = L(Ε·, y) + Ξ» Β· Ξ©(ΞΈ)
Ξ» = regularization strength Β· Ξ©(ΞΈ) = penalty on model weights
| Method | Penalty Ξ©(ΞΈ) | Effect |
|---|---|---|
| L2 (Ridge) | Ξ£ ΞΈα΅’Β² | Shrinks all weights β never zero |
| L1 (Lasso) | Ξ£ |ΞΈα΅’| | Pushes weights to zero β sparse model |
| ElasticNet | Ξ±Β·L1 + (1βΞ±)Β·L2 | Combines both β flexible |
| Dropout | Random neuron off | Neural nets β prevents co-adaptation |
| Early Stopping | Stop at min val loss | Implicit regularization β cheap |
Ξ» tuning: Too small β still overfits. Too large β underfits. Use cross-validation to find the sweet spot.
Data Splits & Cross-Validation
Workflow| Split | Typical % | Purpose |
|---|---|---|
| Training set | 60β80% | Fit model parameters |
| Validation set | 10β20% | Tune hyperparameters, detect overfitting |
| Test set | 10β20% | Final unbiased performance estimate |
k-Fold Cross-Validation
CV Score = (1/k) Β· Ξ£α΅’ Score(fold i)
k=5 or k=10 common Β· Stratified k-Fold preserves class proportions
| Technique | Best For |
|---|---|
| k-Fold CV | General β balanced between bias & variance |
| Stratified k-Fold | Imbalanced classification |
| Leave-One-Out (LOO) | Very small datasets |
| Time-Series Split | Temporal data β no future leakage |
Never let test data touch training or hyperparameter tuning β it's for final evaluation only.
The Machine Learning Workflow
End-to-End- Define the problem β Classification or regression? What is success? Which metric?
- Collect & explore data (EDA) β distributions, missing values, outliers, correlations
- Preprocess & clean β handle nulls, encode categoricals, fix data types
- Feature engineering β create, transform, and select informative features
- Split data β train / validation / test (or k-fold) β split before any fitting
- Choose & train baseline model β simple model first (linear reg, decision tree)
- Evaluate on validation set β accuracy, F1, RMSE, AUC-ROC etc.
- Tune hyperparameters β Grid Search, Random Search, or Bayesian optimization
- Handle overfitting/underfitting β regularization, data augmentation, feature selection
- Train final model β on train + validation; evaluate once on test set
- Interpret & explain β SHAP values, feature importance, partial dependence plots
- Deploy & monitor β serve predictions, watch for data/concept drift
Golden rule: Start simple. A strong baseline + good feature engineering often beats a complex model with raw data.
Hyperparameter Tuning
Optimization| Strategy | How | Best For |
|---|---|---|
| Grid Search | Try all combinations in grid | Small param spaces |
| Random Search | Sample randomly from distributions | Larger param spaces |
| Bayesian Optimization | Model performance surface, sample smartly | Expensive models |
| Halving / Successive | Quickly eliminate bad configs | Large candidate sets |
sklearn β Grid Search CV
from sklearn.model_selection import GridSearchCV params = {'max_depth': [3, 5, 10], 'n_estimators': [50, 100, 200]} gs = GridSearchCV(model, params, cv=5, scoring='f1') gs.fit(X_train, y_train) best = gs.best_params_
Random Search tip: Random Search finds good hyperparameters ~10Γ faster than Grid Search for most problems (Bergstra & Bengio 2012).
Feature Scaling
Preprocessing| Method | Formula | Range | Use When |
|---|---|---|---|
| Min-Max Normalization | (x β min) / (max β min) | [0, 1] | Bounded output needed, no outliers |
| Standardization (Z-score) | (x β ΞΌ) / Ο | ~(β3, 3) | Gaussian-based algos, with outliers |
| Robust Scaler | (x β median) / IQR | Unbounded | Heavy outliers present |
| Log Transform | log(x + 1) | Unbounded | Right-skewed data (income, counts) |
sklearn Scaling Pipeline
from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline pipe = Pipeline([ ('scale', StandardScaler()), ('model', LogisticRegression()) ]) pipe.fit(X_train, y_train)
Critical: Fit scaler on training data only β transform both train and test using training statistics. Never fit on test data.
Imbalanced Data
Data Issue| Strategy | How It Works | When |
|---|---|---|
| Oversampling (SMOTE) | Generate synthetic minority samples | Small dataset, severe imbalance |
| Undersampling | Remove majority class samples | Large dataset |
| Class weights | Penalize minority misclassification more | Quick, no data change |
| Threshold tuning | Adjust decision boundary (default 0.5) | Post-training |
| Ensemble methods | BalancedRF, EasyEnsemble | General-purpose |
class_weight in sklearn
LogisticRegression(class_weight='balanced') RandomForestClassifier(class_weight='balanced') # Or specify manually SVC(class_weight={0: 1, 1: 10})
Avoid accuracy! 95% accuracy on 95/5 split = model predicts majority always. Use F1, AUC-ROC, or Precision-Recall.
Gradient Descent
OptimizationParameter Update Rule
ΞΈ β ΞΈ β Ξ± Β· βL(ΞΈ)
Ξ± = learning rate Β· βL(ΞΈ) = gradient of loss w.r.t. parameters
| Variant | Batch Size | Trade-off |
|---|---|---|
| Batch GD | Full dataset | Stable but slow on large data |
| Stochastic GD (SGD) | 1 sample | Noisy but fast; can escape local minima |
| Mini-batch GD | 32β512 | Best of both β GPU-efficient |
| Optimizer | Key Idea |
|---|---|
| Momentum | Accumulate past gradients β faster convergence |
| RMSProp | Adapt LR per parameter using moving avg of squared grads |
| Adam | Momentum + RMSProp β default choice for most deep learning |
Key ML Terminology
Glossary| Term | Meaning |
|---|---|
| Feature (X) | Input variable used for prediction |
| Label / Target (y) | Output variable the model tries to predict |
| Hyperparameter | Model setting set before training (e.g. learning rate, depth) |
| Parameter (ΞΈ) | Values learned during training (weights, biases) |
| Epoch | One full pass through the entire training dataset |
| Batch | Subset of training data used per gradient update |
| Generalization | Model's ability to perform well on unseen data |
| Data leakage | Test info inadvertently used in training β inflates metrics |
| Inductive bias | Assumptions a model makes about the data (e.g. linearity) |
| Capacity | Model's expressiveness / ability to fit complex functions |
Encoding Categorical Variables
Preprocessing| Encoding | How | When to Use |
|---|---|---|
| Label Encoding | Map categories β integers (0,1,2β¦) | Ordinal variables; tree-based models |
| One-Hot Encoding | Binary column per category | Nominal; linear/distance-based models |
| Ordinal Encoding | Map in meaningful order | Ordinal with natural ranking |
| Target Encoding | Replace category with mean target | High-cardinality categoricals |
| Binary / Hashing | Hash to fixed-length binary | Very high-cardinality, speed critical |
pandas β One-Hot Encoding
import pandas as pd df = pd.get_dummies(df, columns=['color', 'city'], drop_first=True)
Dummy trap: Drop one category per feature (drop_first=True) to avoid perfect multicollinearity in linear models.
Foundational Principles
TheoryNo Free Lunch (NFL) Theorem
No single algorithm is best for all problems
All algorithms perform equally averaged across all possible datasets β domain knowledge matters
Occam's Razor in ML
Prefer the simplest model that fits the data
Simpler models β lower variance β better generalization β complexity must be justified by gain
Universal Approximation Theorem
A neural network with 1 hidden layer can approximate any continuous function
Existence β learnability β doesn't tell us how to find the weights
Practical takeaway: Always try a simple baseline (logistic regression, decision tree) before investing in complex models.
ML Basics Mastery Checklist
Self-AssessmentCore Concepts
Explain the difference between supervised, unsupervised, and reinforcement learning
Distinguish classification from regression with examples
Define features, labels, parameters, and hyperparameters
Explain biasβvariance tradeoff and the error decomposition formula
Identify underfitting and overfitting from train/validation error patterns
Loss Functions
State and apply MSE and MAE for regression problems
State and apply Binary Cross-Entropy for classification
Choose the right loss function for a given task
Explain why log loss penalizes confident wrong predictions heavily
Data & Preprocessing
Split data correctly into train / validation / test sets
Implement k-fold cross-validation and explain why it reduces variance
Apply standardization and min-max normalization correctly
Fit scalers only on training data and transform test data
Encode categorical variables using one-hot or ordinal encoding appropriately
Handle imbalanced classes with SMOTE, class weights, or threshold tuning
Regularization
Explain L1 (Lasso) and L2 (Ridge) regularization and their effects
Tune the regularization strength Ξ» with cross-validation
Apply early stopping to prevent overfitting in iterative models
Optimization & Workflow
Describe gradient descent and the parameter update rule
Compare Batch, SGD, and Mini-batch gradient descent
Explain Adam optimizer and why it's the default in deep learning
Follow the complete ML workflow from problem definition to deployment
Tune hyperparameters with Grid Search or Random Search
Principles
State the No Free Lunch theorem and its practical implication
Apply Occam's Razor β start simple, add complexity only when justified
Detect and prevent data leakage in a pipeline
Choose appropriate evaluation metric for imbalanced classification