What is the maximum value of the derivative of the Sigmoid activation function $\sigma'(z)$?
At $z = 0, \sigma(0) = 0.5$, so $\sigma'(0) = 0.5 * (1 - 0.5) = 0.25$.
What major problem affects deep neural networks using Sigmoid or Tanh activations in hidden layers during backpropagation?
For large |z|, $\sigma'(z) -> 0$. Multiplying small derivatives across many layers causes backpropagated gradients to vanish towards 0.
How does the Rectified Linear Unit (ReLU) activation function f(z) = max(0, z) mitigate the Vanishing Gradient problem?
For positive inputs (z > 0), ReLU derivative is constant 1, preventing gradient decay during backpropagation.
What issue can occur with standard ReLU activations where neurons become permanently inactive and output 0 for all training samples?
If a large negative gradient updates weights such that z < 0 for all inputs, ReLU derivative remains 0 forever, killing the neuron.
How does Leaky ReLU modify standard ReLU to prevent 'dying' neurons?
Leaky ReLU provides a small non-zero slope $\alpha$ for negative inputs, ensuring a non-zero gradient even when z < 0.