Regularization in Machine Learning

What is Regularization? #

Regularization is a technique used to reduce overfitting by adding a penalty to the model’s loss function.

A normal regression model tries to minimize prediction error. A regularized model minimizes prediction error plus a penalty on the model coefficients.

Why Do We Use Regularization? #

A model can become too complex and learn noise instead of the actual relationship in the data. This is called overfitting.

Regularization helps by: #

• Reducing the magnitude of coefficients
• Controlling model complexity
• Reducing overfitting
• Improving generalization on unseen data
• Handling multicollinearity better
• Performing feature selection with Lasso and Elastic Net

Core Idea #

Without regularization:

With regularization:

The model now has two goals:

1. Make predictions accurate.
2. Keep coefficients under control.

Types of Regularization #

1. Ridge Regression #

Ridge Regression uses L2 regularization. It adds the squared coefficient values to the loss function.

How does Ridge work? #

Suppose a model has very large coefficients:

Ridge adds a penalty for these large values. Therefore, the optimization process tries to reduce them.

Important characteristic #

Ridge usually does not make coefficients exactly zero. It mainly shrinks them toward zero.

Implementation #

from sklearn.linear_model import Ridge

model = Ridge(alpha=1)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

When should you use Ridge? #

• Many features are useful.
• Features are highly correlated.
• You want to keep most/all features.
• You mainly want to reduce coefficient magnitude.

2. Lasso Regression #

Lasso uses L1 regularization. Lasso stands for Least Absolute Shrinkage and Selection Operator.

How does Lasso work? #

Lasso penalizes the absolute value of each coefficient. During optimization, some coefficients can become exactly zero.

A coefficient of zero means that feature does not contribute to the final linear prediction of that Lasso model.

Why is this useful? #

Lasso can automatically remove less useful features. Therefore, Lasso performs a form of feature selection.

Implementation #

from sklearn.linear_model import Lasso

model = Lasso(alpha=1)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

When should you use Lasso? #

• You have many features.
• Some features may be irrelevant.
• You want a simpler model.
• You want automatic feature selection.

3. Elastic Net Regression #

Elastic Net combines the advantages of Lasso (L1) and Ridge (L2).

A simplified representation is:

Why was Elastic Net introduced? #

Lasso can perform feature selection, but when features are strongly correlated, it may behave unpredictably by selecting one feature and ignoring another correlated feature.

Ridge handles correlated features well, but generally keeps them. Elastic Net combines both approaches.

Implementation #

from sklearn.linear_model import ElasticNet

model = ElasticNet(
    alpha=0.1,
    l1_ratio=0.5
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

Understanding alpha #

alpha controls the overall strength of regularization.

If alpha is too small #

If alpha is too large #

Understanding l1_ratio #

l1_ratio controls the balance between L1 (Lasso) and L2 (Ridge) in Elastic Net.

Ridge vs Lasso vs Elastic Net #

Which One Should You Use? #

Do We Need Feature Scaling? #

Usually, yes. Regularization penalizes coefficients, so differences in feature scale can affect the penalty.

In real-world projects, use a pipeline:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

model = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge", Ridge(alpha=1))
])

model.fit(X_train, y_train)

How to Find the Best Parameters? #

Do not choose alpha or l1_ratio randomly. Test different values using validation or cross-validation.

from sklearn.linear_model import ElasticNet

for ratio in [0.1, 0.3, 0.5, 0.7, 0.9]:

    model = ElasticNet(
        alpha=0.1,
        l1_ratio=ratio
    )

    model.fit(X_train, y_train)

    y_pred = model.predict(X_test)

    print(
        "L1 Ratio:", ratio,
        "R2:", r2_score(y_test, y_pred),
        "MSE:", mean_squared_error(y_test, y_pred)
    )

In a real project, GridSearchCV or RandomizedSearchCV can be used to search for good hyperparameters systematically.

Common Mistakes #