---
title: Regularization in Machine Learning
description: "MACHINE LEARNING A practical guide to Ridge, Lasso and Elastic Net — why regularization is needed, how it works, and when to use each method. ## What is Regularization? **Regularization** is a technique used to reduce **overfitting** by adding a…"
url: "https://csestudy247.com/docs/regularization-in-machine-learning/"
updated: "2026-08-16"
category: ML
---

# Regularization in Machine Learning

MACHINE LEARNING

A practical guide to Ridge, Lasso and Elastic Net — why regularization is needed, how it works, and when to use each method.

## What is Regularization?

**Regularization** is a technique used to reduce **overfitting** by adding a penalty to the model's loss function.

A normal regression model tries to minimize prediction error. A regularized model minimizes prediction error **plus a penalty on the model coefficients.**

  Loss = MSE + Regularization Penalty   **Simple idea:** Regularization tells the model: *"Fit the data well, but don't use unnecessarily large coefficients."*

## Why Do We Use Regularization?

A model can become too complex and learn noise instead of the actual relationship in the data. This is called **overfitting**.

  Complex Model → Large Coefficients → Overfitting → Poor Generalization

### Regularization helps by:

- Reducing the magnitude of coefficients
- Controlling model complexity
- Reducing overfitting
- Improving generalization on unseen data
- Handling multicollinearity better
- Performing feature selection with Lasso and Elastic Net

## Core Idea

Without regularization:

  Loss = MSE

With regularization:

  Loss = MSE + Penalty

The model now has two goals:

1. Make predictions accurate.
2. Keep coefficients under control.

  **Important:** Regularization does not mean "make every coefficient zero". It means adding a cost for large coefficients.

## Types of Regularization

### Ridge

 L2 REGULARIZATION

Penalizes squared coefficients.

 **Main effect:**

Usually shrinks coefficients without removing features.

### Lasso

 L1 REGULARIZATION

Penalizes absolute coefficient values.

 **Main effect:**

Can make coefficients exactly zero.

### Elastic Net

 L1 + L2

Combines Lasso and Ridge.

 **Main effect:**

Coefficient shrinkage + feature selection.

## 1. Ridge Regression

Ridge Regression uses **L2 regularization**. It adds the squared coefficient values to the loss function.

  Loss = MSE + α Σ(w²)

### How does Ridge work?

Suppose a model has very large coefficients:

  w1 = 500
 w2 = -800
 w3 = 1200

Ridge adds a penalty for these large values. Therefore, the optimization process tries to reduce them.

  Large Coefficients → L2 Penalty → Smaller Coefficients

### Important characteristic

Ridge usually **does not make coefficients exactly zero**. It mainly shrinks them toward zero.

### Implementation

```
from sklearn.linear_model import Ridge

model = Ridge(alpha=1)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)
```

### When should you use Ridge?

- Many features are useful.
- Features are highly correlated.
- You want to keep most/all features.
- You mainly want to reduce coefficient magnitude.

  **Remember:** Ridge = **Reduce / Shrink**

## 2. Lasso Regression

Lasso uses **L1 regularization**. Lasso stands for **Least Absolute Shrinkage and Selection Operator**.

  Loss = MSE + α Σ|w|

### How does Lasso work?

Lasso penalizes the absolute value of each coefficient. During optimization, some coefficients can become exactly zero.

  Before Lasso:
 age = 120
 bmi = 450
 bp = 80
 s1 = 90

 After Lasso:
 age = 0
 bmi = 413
 bp = 35
 s1 = 0

A coefficient of zero means that feature does not contribute to the final linear prediction of that Lasso model.

### Why is this useful?

Lasso can automatically remove less useful features. Therefore, Lasso performs a form of **feature selection**.

### Implementation

```
from sklearn.linear_model import Lasso

model = Lasso(alpha=1)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)
```

### When should you use Lasso?

- You have many features.
- Some features may be irrelevant.
- You want a simpler model.
- You want automatic feature selection.

  **Remember:** Lasso = **Select / Remove**

## 3. Elastic Net Regression

Elastic Net combines the advantages of **Lasso (L1)** and **Ridge (L2)**.

  Elastic Net = L1 + L2

A simplified representation is:

  Loss = MSE + α \[L1 + L2\]

### Why was Elastic Net introduced?

Lasso can perform feature selection, but when features are strongly correlated, it may behave unpredictably by selecting one feature and ignoring another correlated feature.

Ridge handles correlated features well, but generally keeps them. Elastic Net combines both approaches.

  Lasso + Ridge → Elastic Net

### Implementation

```
from sklearn.linear_model import ElasticNet

model = ElasticNet(
    alpha=0.1,
    l1_ratio=0.5
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)
```

  **Remember:** Elastic Net = **Both**

## Understanding `alpha`

`alpha` controls the **overall strength of regularization**.

   **Small α** Weak regularization   **Medium α** Moderate regularization   **Large α** Strong regularization

### If alpha is too small

  Weak Penalty → Complex Model → Possible Overfitting

### If alpha is too large

  Strong Penalty → Very Small Coefficients → Possible Underfitting   **Important:** The largest alpha is not automatically the best alpha. Choose it based on validation or cross-validation performance.

## Understanding `l1_ratio`

`l1_ratio` controls the balance between **L1 (Lasso)** and **L2 (Ridge)** in Elastic Net.

   **0.0** Pure Ridge   **0.25** Mostly Ridge   **0.50** Balanced   **0.75** Mostly Lasso   **1.0** Pure Lasso

### Higher l1_ratio

More Lasso behavior.

More pressure toward zero coefficients.

### Lower l1_ratio

More Ridge behavior.

More coefficient shrinking.

    **Easy Memory:**
 `alpha` = How strong is the penalty?
 `l1_ratio` = Lasso or Ridge balance?

## Ridge vs Lasso vs Elastic Net

| Property | Ridge | Lasso | Elastic Net |
| --- | --- | --- | --- |
| Regularization | L2 | L1 | L1 + L2 |
| Shrinks coefficients | Yes | Yes | Yes |
| Can make coefficient zero | Usually No | Yes | Yes |
| Feature selection | No | Yes | Yes |
| Handles correlated features | Good | Can be unstable | Good |
| Main parameters | alpha | alpha | alpha + l1_ratio |

## Which One Should You Use?

### Use Ridge when...

- Most features are useful.
- Features are correlated.
- You want to keep all features.

### Use Lasso when...

- You have many features.
- Some features may be irrelevant.
- You want automatic feature selection.

### Use Elastic Net when...

- You have many features.
- Features are correlated.
- You also want feature selection.

## Do We Need Feature Scaling?

Usually, **yes**. Regularization penalizes coefficients, so differences in feature scale can affect the penalty.

In real-world projects, use a pipeline:

```
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

model = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge", Ridge(alpha=1))
])

model.fit(X_train, y_train)
```

  The Scikit-learn Diabetes dataset is already scaled, so it is convenient for learning regularized regression.

## How to Find the Best Parameters?

Do not choose `alpha` or `l1_ratio` randomly. Test different values using validation or cross-validation.

```
from sklearn.linear_model import ElasticNet

for ratio in [0.1, 0.3, 0.5, 0.7, 0.9]:

    model = ElasticNet(
        alpha=0.1,
        l1_ratio=ratio
    )

    model.fit(X_train, y_train)

    y_pred = model.predict(X_test)

    print(
        "L1 Ratio:", ratio,
        "R2:", r2_score(y_test, y_pred),
        "MSE:", mean_squared_error(y_test, y_pred)
    )
```

In a real project, **GridSearchCV** or **RandomizedSearchCV** can be used to search for good hyperparameters systematically.

## Common Mistakes

  **1. Higher alpha is always better**

False. Too much regularization can cause underfitting.

   **2. Ridge removes features**

Ridge normally shrinks coefficients but does not force them to zero.

   **3. Lasso and Ridge do the same thing**

Lasso uses L1 and can perform feature selection. Ridge uses L2 and mainly shrinks coefficients.

   **4. Confusing alpha and l1_ratio**

alpha controls regularization strength. l1_ratio controls the L1/L2 balance.

## Quick Revision

   RIDGE **L2**

Shrinks coefficients

   LASSO **L1**

Can remove features

   ELASTIC NET **L1 + L2**

Combines both

    **One-line memory trick:**

 Ridge = **Shrink** | Lasso = **Select** | Elastic Net = **Both**