How does Momentum SGD update velocity v_t with momentum factor \beta and learning rate eta?
Momentum updates velocity v_t = \beta v_{t-1} + eta g_t and updates parameter theta_t = theta_{t-1} - $v_t$.
What is the effect of L2 Weight Decay on weight parameter updates during Gradient Descent?
$With L2 loss penalty (\lambda/2)||w||^2, gradient becomes g + \lambda w, giving update w <- (1 - eta \lambda) w - eta g.$
Which technique prevents Exploding Gradients in Recurrent or Deep Neural Networks by thresholding the gradient norm?
Gradient Clipping rescales gradient g <- g * (threshold / ||g||) if ||g|| > threshold, preventing huge gradient steps.
A neural network hidden layer receives input x and computes z = W x + b. If x has dimension 10 and z has dimension 20, what is the dimension of weight matrix W?
$To map 10-dim input x to 20-dim output z via z = W x + b, weight matrix W must have size 20 x 10.$
Why can compositions of purely LINEAR hidden activation functions in MLPs NOT model non-linear boundaries?
$W2 (W1 x + b1) + b2 = (W2 W1) x + (W2 b1 + b2) = W_{eff} x + b_{eff}, collapsing multi-layer network into a single linear model.$