Which weight initialization sets weight variance Var(W) = 2 / ($n_{in}$ + $n_{out}$) (designed for Sigmoid and Tanh activations)?
Xavier initialization maintains variance of activations and backpropagated gradients across layers for linear/Sigmoid/Tanh networks.
In Dropout regularization during training with dropout probability p, how does INVERTED DROPOUT adjust active neuron activations?
Inverted dropout divides kept activations by (1 - p) during training, leaving test-time inference untouched.
Batch Normalization normalizes mini-batch activations x across the batch dimension to zero mean and unit variance, then applies what learnable transformation?
Batch Normalization introduces learnable scale $\gamma$ and shift $\beta parameters$ to preserve model representation capacity.
Which optimization algorithm incorporates Exponential Moving Averages of BOTH past gradients (1st moment / momentum) and past squared gradients (2nd moment)?
Adam combines momentum ($m_{t} = beta1 m_{t-1} + (1-beta1) g_{t}$) and RMSprop ($v_{t} = beta2 v_{t-1} + (1-beta2) g_{t}^2$) with bias correction.
What limitation of the AdaGrad optimizer does RMSprop fix?
AdaGrad accumulates squared gradients indefinitely, causing learning rate to vanish. RMSprop replaces the sum with an exponential moving average.