Regularization Techniques

In the previous lesson, we learned about deep learning and how neural networks become powerful by adding more layers.

However, increasing model complexity introduces a serious problem: overfitting.

This lesson explains Regularization, a set of techniques used to control model complexity and improve generalization.

What Is Overfitting?

Overfitting happens when a model learns the training data too well.

Instead of learning general patterns, the model memorizes noise and small details.

As a result, the model performs very well on training data but poorly on new, unseen data.

This problem is especially common in deep learning models.

Why Regularization Is Needed

Neural networks have many parameters.

With enough capacity, they can fit almost any dataset perfectly.

Regularization limits the model’s freedom so that it learns meaningful patterns instead of memorizing data.

Think of regularization as adding discipline to learning.

Regularization in Our Dataset Context

Using the Dataplexa ML dataset, a deep model could memorize specific customer profiles.

This would cause incorrect predictions for new loan applicants.

Regularization forces the model to focus on general financial behavior rather than individual cases.

L2 Regularization (Weight Decay)

L2 regularization adds a penalty for large model weights.

By discouraging large weights, the model becomes smoother and less sensitive to small changes in input.

This is one of the most widely used regularization techniques.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    penalty="l2",
    C=1.0,
    max_iter=1000
)

model.fit(X_train, y_train)

A smaller value of C means stronger regularization.

L1 Regularization (Feature Selection Effect)

L1 regularization also penalizes large weights, but in a different way.

It can shrink some weights to zero, effectively removing less important features.

This makes L1 useful for feature selection.

model = LogisticRegression(
    penalty="l1",
    solver="liblinear",
    C=1.0,
    max_iter=1000
)

model.fit(X_train, y_train)

Dropout Regularization (Deep Learning)

Dropout is a regularization technique used mainly in neural networks.

During training, random neurons are temporarily disabled.

This prevents neurons from becoming too dependent on each other.

Dropout improves robustness and generalization.

Early Stopping

Early stopping monitors validation performance during training.

When performance stops improving, training is stopped.

This prevents the model from overfitting the training data.

Choosing the Right Regularization

There is no single best regularization technique.

The choice depends on:

• Dataset size • Model complexity • Noise level

Regularization strength is usually tuned using validation data.

Mini Practice

Train a model with and without regularization and compare validation performance.

Observe how regularization affects generalization.

Exercises

Quick Quiz

In the next lesson, we study Loss Functions, which guide how models learn during training.