GATE DA Supervised Learning – Data Science & AI | Set 1

The correct answer is A \(w = (X^T X)^{-1} X^T y\).

Step-by-Step Derivation

1. Formulate the Cost Function:
In Ordinary Least Squares (OLS) regression, the goal is to find a weight vector \(w\) that minimizes the sum of squared errors between the actual targets \(y\) and the model's predictions \(\hat{y} = Xw\). The loss function \(J(w)\) is expressed in matrix form as:
\(J(w)=\frac{1}{2}\|{}Xw-y\|{}^{2}=\frac{1}{2}(Xw-y)^{T}(Xw-y)\)
2. Expand the Matrix Expression:
Distribute the matrix transpose operations inside the parenthesis:
\(J(w)=\frac{1}{2}(w^{T}X^{T}-y^{T})(Xw-y)\)
\(J(w)=\frac{1}{2}(w^{T}X^{T}Xw-w^{T}X^{T}y-y^{T}Xw+y^{T}y)\)Since \(w^{T}X^{T}y\) is a scalar value, it is equal to its own transpose \((w^T X^T y)^T = y^T X w\). Simplifying the middle terms gives:
\(J(w)=\frac{1}{2}(w^{T}X^{T}Xw-2w^{T}X^{T}y+y^{T}y)\)
3. Compute the Gradient:
To find the value of \(w\) that minimizes the error, calculate the matrix derivative (gradient) of the scalar function \(J(w)\) with respect to the vector \(w\):
\(\nabla _{w}J(w)=X^{T}Xw-X^{T}y\)
4. Solve the Normal Equation:
Set the gradient equal to zero to identify the stationary point:
\(X^{T}Xw-X^{T}y=0\)
\(X^{T}Xw=X^{T}y\)Assuming that the cross-product matrix \(X^{T}X\) is non-singular and invertible, multiply both sides from the left by \((X^T X)^{-1}\) to isolate \(w\):
\(w=(X^{T}X)^{-1}X^{T}y\)

Original solution basis:
Logistic Regression models probabilities P(y=1|x) = sigmoid(w^{T} x) and minimizes negative log-likelihood, which is Binary Cross-Entropy loss.

Deep Dive
For each training example with label yᵢ ∈ {0,1} and predicted probability pᵢ, binary cross-entropy is -[yᵢ $\log(pᵢ) + (1-yᵢ)\log(1-pᵢ)]$. The average of this quantity is the negative log-likelihood of the Bernoulli model. Minimizing it is therefore equivalent to maximizing the likelihood of the observed labels. Hinge loss is associated with margin-based classifiers such as SVMs, not standard logistic regression.

Key Points to Remember:
• Binary cross-entropy = negative Bernoulli log-likelihood.
• Logistic regression predicts probabilities using the sigmoid function.
• Use log loss for standard probabilistic binary logistic regression.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
d/dw [(1/2n)(Xw - y)^T (Xw - y)] = (1/n) X^{T} (Xw - y).

Deep Dive
Let r = Xw - y be the n-dimensional residual vector. The loss is L(w) = (1/2n)rᵀr. Since ∂r/∂w = X, the chain rule gives ∇wL = (1/n)Xᵀr = (1/n)Xᵀ(Xw-y). The gradient is a d-dimensional column vector, matching the dimension of w. The factor 1/2 cancels the factor 2 produced by differentiating the squared norm.

Key Points to Remember:
• For L = (1/2n)||Xw-y||², ∇L = Xᵀ(Xw-y)/n.
• The gradient has the same dimension as the parameter vector w.
• The transpose is essential for converting the n-dimensional residual into a d-dimensional gradient.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Differentiating ||Xw - y||^2 + $\lambda ||w||^2 w.r.t w yields 2 X^{T}(Xw - y) + 2 \lambda w = 0 => (X^{T} X + \lambda I)$$w = X^{T} y$.

Deep Dive
Differentiate J(w) = (Xw-y)ᵀ(Xw-y) + $\lambda wᵀw$. Its gradient is 2Xᵀ(Xw-y) + 2$\lambda w$. Setting the gradient to zero and dividing by 2 gives XᵀXw - Xᵀy + $\lambda w = 0$. Rearranging yields (XᵀX + \lambda I)w = Xᵀy, hence the stated solution. For \lambda >0, XᵀX+\lambda I is positive definite and invertible, even when XᵀX is singular.

Key Points to Remember:
• Ridge adds \lambda ||w||² to the squared-error objective.
• The normal equation becomes (XᵀX+$\lambda I)w=Xᵀy$.
• \lambda >0 stabilizes the solution; the intercept is often excluded from penalization in practice.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
L1 norm ||w||_1 = sum |$w_i$| has non-differentiable corners along coordinate axes where the elliptical loss contours frequently intersect first.

Deep Dive
The L1 penalty is $\lambda \Sigma ⱼ|wⱼ|$. Unlike the smooth circular contours associated with an L2 constraint, the L1 constraint region has corners on coordinate axes. When a loss contour first touches this region, the contact often occurs at a corner, where one or more coordinates are exactly zero. This geometric property explains why Lasso can perform feature selection. L1 regularization is convex, not non-convex.

Key Points to Remember:
• Lasso minimizes data loss plus $\lambda ||w||₁$.
• The L1 geometry encourages exact zero coefficients and sparse models.
• L1 is convex but non-differentiable at zero; subgradient methods handle this.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.