Gradient Boosting

Gradient Boosting is an ensemble machine learning technique that builds models sequentially. Unlike Random Forest, where trees are built independently, Gradient Boosting builds each new tree to correct the mistakes made by the previous ones.

Instead of asking many experts at once, Gradient Boosting learns step by step, improving itself at every stage.

Real-World Connection

Think of a student preparing for exams. After the first test, the student reviews mistakes, studies weak topics, and improves in the next test. Gradient Boosting works the same way by focusing more on the errors made earlier.

How Gradient Boosting Works

The learning process happens in stages:

• The first model makes simple predictions
• Errors from that model are calculated
• The next model focuses on correcting those errors
• This process continues until performance improves

Key Idea Behind Gradient Boosting

Each new model minimizes the loss function using gradient descent. That is why the word gradient appears in the name. The model moves step by step in the direction that reduces errors.

Gradient Boosting for Regression

from sklearn.ensemble import GradientBoostingRegressor
import numpy as np

X = np.array([[1],[2],[3],[4],[5]])
y = np.array([2,4,6,8,10])

model = GradientBoostingRegressor(
    n_estimators=100,
    learning_rate=0.1,
    random_state=42
)

model.fit(X, y)
print(model.predict([[6]]))
  

The model predicts values by combining many weak learners. Each learner slightly improves the prediction by reducing previous errors.

Understanding the Code

The n_estimators parameter controls how many trees are built. The learning_rate decides how much each tree contributes. Smaller learning rates improve accuracy but require more trees.

Gradient Boosting for Classification

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    random_state=42
)

model.fit(X, y)
print(model.score(X, y))
  

For classification, Gradient Boosting uses probability-based predictions and improves class separation at each stage.

Advantages of Gradient Boosting

• High predictive accuracy
• Works well with complex data
• Handles bias effectively
• Flexible loss functions

Limitations of Gradient Boosting

• Slower training
• Sensitive to noise
• Needs careful tuning

Gradient Boosting vs Random Forest

Random Forest reduces variance by averaging many independent trees, while Gradient Boosting reduces bias by learning from mistakes. Gradient Boosting often performs better but requires more tuning.

Practice Questions

Quick Quiz

Coming up next: XGBoost — an optimized and scalable version of Gradient Boosting.