GATE DA Supervised Learning – Data Science & AI | Set 1

Original solution basis:
Collinear features cause X^{T} X to have near-zero determinant, making (X^{T} X)^{-1} unstable and coefficient estimates volatile.

Deep Dive
If columns of X are strongly correlated, one column is close to a linear combination of others. Then XᵀX becomes ill-conditioned or singular. Its inverse, when it exists, can amplify small data perturbations, causing large changes in estimated coefficients. Exact collinearity makes XᵀX singular; near-collinearity makes it numerically unstable.

Key Points to Remember:
• Multicollinearity makes XᵀX ill-conditioned or singular.
• Coefficient estimates can have high variance and be unstable.
• Prediction may remain reasonable even when individual coefficients are poorly determined.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Adding positive \lambda to diagonal eigenvalues guarantees matrix invertibility and conditions numeric stability.

Deep Dive
The matrix XᵀX is positive semidefinite. Adding \lambda I with \lambda >0 shifts each eigenvalue upward by $\lambda , making every eigenvalue strictly positive$. Thus XᵀX+$\lambda I$ is positive definite and invertible. Ridge regression stabilizes coefficient estimation by shrinking weights, though it introduces bias.

Key Points to Remember:
• Ridge replaces XᵀX by XᵀX+$\lambda I$.
• For $\lambda >0$, the shifted matrix is positive definite.
• Ridge stabilizes multicollinear estimates by accepting some bias.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
For separable data, scaling w -> infinity drives sigmoid(w^{T} x) to 0 or 1, driving log-likelihood to 0. Regularization prevents weight explosion.

Deep Dive
With perfectly linearly separable data, logistic regression's likelihood can keep improving as the coefficient vector is scaled to larger magnitude in the separating direction. Predicted probabilities approach zero or one for the observed labels, but no finite coefficient vector may attain the supremum. Consequently, unregularized MLE estimates can diverge rather than converge to a finite optimum.

Key Points to Remember:
• Complete separation can prevent existence of a finite logistic-regression MLE.
• Coefficient magnitudes may grow without bound while likelihood approaches its supremum.
• Regularization or bias-reduced estimation can address this issue.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
H(S) = - 0.5 log2(0.5) - 0.5 log2(0.5) = 0.5 + 0.5 = 1.0 bit (maximum impurity for binary classification).

Deep Dive
For p₊=p₋=0.5, H=-0.5log₂0.5-0.5log₂0.5. Since log₂0.5=-1, each term contributes 0.5 bit, giving H=1 bit. This is the maximum entropy for a binary node because uncertainty is greatest when the classes are equally likely.

Key Points to Remember:
• Binary entropy is maximum at a 50:50 class split.
• H=1 bit for p₊=p₋=0.5.
• A pure binary node has entropy 0.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
For a pure node with p1 = 1, p2 = 0: Gini = 1 - (1^2 + 0^2) = 0.0.

Deep Dive
For a pure node, one class has probability 1 and every other class has probability 0. Therefore Gini=1-(1²+0²)=0. There is no class mixture in the node, so a randomly selected sample cannot be misclassified by predicting the only class present.

Key Points to Remember:
• Gini impurity = 1-$\Sigma pᵢ²$.
• A pure node has Gini impurity 0.
• For binary classes, the maximum Gini impurity is 0.5 at a 50:50 split.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.