What is the mathematical formulation of the Softmax function for output neuron i given logits z = [$z_1$, ..., $z_C$]?
$Softmax converts unnormalized logit scores z into a valid probability distribution p_i = e^{z_i} / sum e^{z_k}.$
When using Softmax activation at the output layer with Categorical Cross-Entropy loss, what is the simplified derivative of loss L with respect to output logit $z_i$?
The combination of Softmax and Cross-Entropy loss yields the elegant gradient dL/dz_i = p_i - $y_i$.
In Backpropagation, the error \delta \delta^{(l)} at layer l is computed from layer l+1 using which mathematical operation?
The backpropagation error equation is $\delta^{(l)} = ((W^{(l+1)})^{T} \delta^{(l+1)}) (elementwise product) f'(z^{(l)})$.
What happens if all weights in a feedforward neural network hidden layer are initialized to ZERO?
Zero initialization causes all hidden neurons to be symmetric (same activations and gradients), preventing them from learning distinct features.
Which weight initialization method sets initial weight variance Var(W) = 2 / n_{in} (designed specifically for ReLU activations)?
He initialization sets Var(W) = 2 / n_{in} to account for ReLU zeroing out half of the activations.