GATE DA Supervised Learning – Data Science & AI | Set 1

Original solution basis:
Differentiating Lagrangian w.r.t w yields $w = \sum_{i=1}^{N} alpha_i y_i x_i$.

Deep Dive
For the linear SVM, stationarity of the Lagrangian with respect to w gives w-$\Sigma ᵢ\alpha ᵢyᵢxᵢ=0$. Rearranging yields w=$\Sigma ᵢ\alpha ᵢyᵢxᵢ$. Only examples with $\alpha ᵢ>0 contribute$, which is why the support vectors determine the separating hyperplane.

Key Points to Remember:
• Dual representation: w=$\Sigma ᵢ\alpha ᵢyᵢxᵢ$.
• Only nonzero $\alpha ᵢ terms contribute$.
• The labels yᵢ are included in the weighted sum.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Soft-margin SVM minimizes (1/2)||w||^2 + C sum max(0, 1 - y_i(w^{T} $x_i$ + b)), where max(0, 1 - y f(x)) is Hinge loss.

Deep Dive
For a signed label y∈{-1,+1} and score f(x)=wᵀx+b, hinge loss is max(0,1-yf(x)). A correctly classified point outside the margin has zero hinge loss; a point inside the margin or misclassified has positive loss. Soft-margin SVM adds these penalties to the norm regularizer.

Key Points to Remember:
• Hinge loss = max(0,1-yf(x)).
• Zero loss requires yf(x)$\ge 1, i.e$. correct classification with margin.
• Soft-margin objective combines (1/2)||w||² with C times total hinge loss.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Expected test MSE decomposes into (E[$f_{hat}$] - $f)^2 + E$[(f_{hat} - E[$f_{hat}$]$)^2$] + $\sigma^2 = Bias^2 + Variance + Irreducible Error$.

Deep Dive
For squared-error regression, expected prediction error can be decomposed into squared bias, variance, and irreducible noise: E[(Y-f̂(X))²]=Bias²+Variance+$\sigma ²$. Bias measures systematic error of the model's average prediction; variance measures sensitivity to the training sample; irreducible noise is inherent randomness in the data and cannot be eliminated by model choice.

Key Points to Remember:
• Expected test MSE = Bias² + Variance + irreducible noise.
• High model flexibility often lowers bias but can raise variance.
• Irreducible noise cannot be removed by fitting a more complex model.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Underfitting occurs when a model is overly simplistic (high bias) and unable to capture underlying data trends on both training and test sets.

Deep Dive
An overly simple model cannot capture the underlying relationship, so it makes systematic errors even on the training data. This is high bias. Because the fitted model changes relatively little across different training samples, its variance is usually low. The result is underfitting and poor performance on both training and unseen data.

Key Points to Remember:
• Underfitting is associated with high bias.
• Simple models often have relatively low variance.
• Remedies include increasing model capacity or adding informative features.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Overfitting occurs when a model captures noise in training data (high variance, low training error), failing to generalize to new data.

Deep Dive
An overly flexible model can fit random fluctuations and noise in the training sample. It may achieve very low training error but change substantially when trained on a different sample, which is high variance. This leads to a gap between training and test performance.

Key Points to Remember:
• Overfitting is associated with high variance and often low bias.
• Training performance can be excellent while test performance is poor.
• Regularization, pruning, more data, and early stopping can reduce overfitting.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.