Feature Selection Techniques

Feature Selection is the process of choosing the most relevant input features for a machine learning model. Instead of creating new features, feature selection focuses on removing unnecessary or weak features.

In real-world AI systems, using fewer but meaningful features often produces better results than using many irrelevant ones.

Why Feature Selection Is Important

Not all features help a model learn. Some features add noise, increase complexity, and slow down training without improving accuracy.

Feature selection helps by:

• Reducing overfitting
• Improving model performance
• Reducing training time
• Making models easier to interpret

Real-World Example

Suppose you are predicting employee salary. Your dataset may include name, employee ID, department, years of experience, and education.

Features like employee ID or name do not help prediction and should be removed. Feature selection identifies and keeps only useful information.

Types of Feature Selection

Feature selection methods are broadly grouped into three categories:

• Filter Methods
• Wrapper Methods
• Embedded Methods

Filter Methods

Filter methods select features based on statistical properties without training a model. They are fast and simple.

Common examples include correlation and variance threshold.

Correlation-Based Feature Selection

import pandas as pd

data = {
    'Age': [25, 30, 35, 40],
    'Experience': [1, 3, 5, 7],
    'Salary': [30000, 50000, 70000, 90000]
}

df = pd.DataFrame(data)
print(df.corr())
  

Highly correlated features may be redundant. One of them can often be removed.

Wrapper Methods

Wrapper methods evaluate feature subsets by training a model and measuring performance. They provide better results but are slower.

A common wrapper technique is Recursive Feature Elimination (RFE).

Recursive Feature Elimination Example

from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[25, 1], [30, 3], [35, 5], [40, 7]])
y = np.array([30000, 50000, 70000, 90000])

model = LinearRegression()
selector = RFE(model, n_features_to_select=1)
selector.fit(X, y)

print(selector.support_)
  

The output shows which feature is selected as most important.

Embedded Methods

Embedded methods perform feature selection during model training. They balance performance and efficiency.

Tree-based models naturally perform embedded feature selection.

Feature Importance with Random Forest

from sklearn.ensemble import RandomForestRegressor

X = [[25, 1], [30, 3], [35, 5], [40, 7]]
y = [30000, 50000, 70000, 90000]

model = RandomForestRegressor()
model.fit(X, y)

print(model.feature_importances_)
  

Higher importance values indicate stronger influence on predictions.

Feature Selection vs Feature Engineering

• Feature Engineering: Creates new features
• Feature Selection: Chooses the best existing features

When Feature Selection Is Most Useful

• High-dimensional datasets
• Models prone to overfitting
• Interpretable AI systems
• Limited computational resources

Practice Questions

Quick Quiz

Coming up next: Model Evaluation Metrics — measuring how well AI models perform.