Gradient Boosting

Gradient Boosting is an advanced ensemble learning technique that builds models sequentially. Unlike Random Forest, where all trees are independent, Gradient Boosting trains each new model to correct the mistakes made by the previous one.

This lesson explains why Gradient Boosting was introduced, how it works step by step, and how it is used in real-world machine learning systems.

Why Gradient Boosting Was Created

Random Forest improves accuracy by averaging many independent trees, but it does not actively focus on mistakes. Some data points are consistently predicted wrong and remain wrong.

Gradient Boosting was designed to fix this problem by learning from errors.

• First model makes predictions
• Second model focuses on mistakes of the first
• Each new model improves the previous one

What Is Gradient Boosting?

Gradient Boosting is an ensemble technique where models are added one by one, and each model is trained on the residual errors of the previous models.

The final prediction is the weighted sum of all models.

Real-World Example

Imagine a teacher checking exam papers. The first review catches most mistakes. The second review focuses only on missed errors. The third review focuses on what was still missed.

Each review improves the final score. This is exactly how Gradient Boosting works.

How Gradient Boosting Works

The process follows these steps:

• Start with a simple model
• Calculate prediction errors
• Train a new model to reduce those errors
• Repeat until errors are minimized

The term gradient comes from optimization — the algorithm moves in the direction that reduces loss.

Simple Gradient Boosting Example

Below is a basic example using Gradient Boosting for classification.

from sklearn.ensemble import GradientBoostingClassifier

# Sample data
X = [[20, 30000], [25, 40000], [30, 50000], [35, 65000], [45, 85000]]
y = [0, 0, 0, 1, 1]

# Create model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)

# Train model
model.fit(X, y)

# Predict
prediction = model.predict([[32, 60000]])
print(prediction)
  

The model predicts approval by gradually learning from previous errors instead of making independent decisions.

Key Parameters Explained

Gradient Boosting performance depends heavily on tuning.

• n_estimators: Number of trees added sequentially
• learning_rate: Controls how much each tree contributes
• max_depth: Depth of individual trees

Advantages of Gradient Boosting

• High predictive accuracy
• Handles complex patterns well
• Strong performance on structured data

Limitations of Gradient Boosting

• More sensitive to noise
• Slower training compared to Random Forest
• Requires careful tuning

Practice Questions

Quick Quiz

Coming up next: XGBoost — an optimized and production-ready version of Gradient Boosting.