Dimensionality Reduction

In real-world datasets, we often deal with a large number of features. While more features may seem helpful, they can actually slow down models, increase noise, and reduce performance. Dimensionality Reduction is the process of reducing the number of input features while preserving as much useful information as possible.

This lesson introduces two widely used techniques: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA).

Real-World Connection

Imagine summarizing a long book into a short summary. You remove less important details but keep the main ideas. Dimensionality reduction does the same for data — it compresses information while keeping the most meaningful patterns.

Why Dimensionality Reduction Is Important

• Improves model performance
• Reduces overfitting
• Speeds up training time
• Makes data easier to visualize

Principal Component Analysis (PCA)

PCA is an unsupervised technique that transforms data into a new coordinate system. The new axes, called principal components, capture the maximum variance in the data.

• First component captures most variance
• Second component captures next most variance
• Components are orthogonal (independent)

PCA Example (Python)

from sklearn.decomposition import PCA
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)

print(X_reduced.shape)
  

Understanding the Output

The dataset originally had four features. After applying PCA, it has been reduced to two features while preserving most of the original variance. This makes visualization and modeling easier.

Linear Discriminant Analysis (LDA)

LDA is a supervised dimensionality reduction technique. Unlike PCA, it considers class labels and aims to maximize the separation between classes.

• Uses labeled data
• Maximizes class separability
• Often used before classification

LDA Example (Python)

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X, y)

print(X_lda.shape)
  

PCA vs LDA

• PCA: Unsupervised, focuses on variance
• LDA: Supervised, focuses on class separation
• PCA ignores labels, LDA uses labels

When to Use Which

• Use PCA for visualization and noise reduction
• Use LDA for classification tasks

Practice Questions

Quick Quiz

Coming up next: Feature Engineering — transforming raw data into powerful inputs for models.