Hyperparameter Tuning

Hyperparameter Tuning is the process of finding the best configuration settings for a machine learning model before training begins. These settings control how the model learns, not what it learns from the data.

Choosing the right hyperparameters can significantly improve model performance, while poor choices can lead to underfitting or overfitting.

What Are Hyperparameters?

Hyperparameters are values that are set manually before training. They are different from model parameters, which are learned automatically during training.

• Learning rate
• Number of trees
• Maximum depth
• Regularization strength

Real-World Connection

Think of hyperparameters like the settings on a cooking stove. The ingredients stay the same, but changing the flame intensity and cooking time changes the final dish.

Why Hyperparameter Tuning Matters

Default hyperparameter values rarely produce the best results. Tuning allows the model to adapt better to the problem and data.

• Improves accuracy
• Reduces overfitting
• Optimizes training efficiency
• Produces more stable models

Manual Hyperparameter Tuning

The simplest approach is manual tuning, where you try different values and observe performance changes.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)

for depth in [2, 4, 6]:
    model = DecisionTreeClassifier(max_depth=depth)
    score = cross_val_score(model, X, y, cv=5).mean()
    print(depth, score)
  

Here, a depth of 4 gives the best average accuracy.

Grid Search

Grid Search automatically tests all combinations of specified hyperparameter values and selects the best one.

from sklearn.model_selection import GridSearchCV

param_grid = {
    'max_depth': [2, 4, 6],
    'min_samples_split': [2, 5, 10]
}

grid = GridSearchCV(
    DecisionTreeClassifier(),
    param_grid,
    cv=5
)

grid.fit(X, y)

print(grid.best_params_)
  

Grid Search evaluates every possible combination and selects the best-performing configuration.

Random Search

Random Search tests a random subset of hyperparameter combinations. It is faster and often nearly as effective as Grid Search.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'max_depth': randint(2, 10),
    'min_samples_split': randint(2, 10)
}

random_search = RandomizedSearchCV(
    DecisionTreeClassifier(),
    param_dist,
    n_iter=10,
    cv=5
)

random_search.fit(X, y)

print(random_search.best_params_)
  

Random Search is useful when the search space is large and time is limited.

Grid Search vs Random Search

• Grid Search is exhaustive but slow
• Random Search is faster and scalable
• Both use cross-validation internally

Practice Questions

Quick Quiz

Coming up next: Regularization — controlling model complexity to prevent overfitting.