What is the mathematical range and zero-centering property of the Hyperbolic Tangent (tanh) activation function?
Tanh maps real numbers to (-1, 1) and is zero-centered, which helps stabilize gradient updates during neural network training compared to standard sigmoid.
What is the mathematical definition of the Rectified Linear Unit (ReLU) activation function?
The ReLU activation function outputs z if z > 0, and 0 otherwise: f(z) = max(0, z). Its constant derivative of 1 for positive inputs mitigates vanishing gradients.
What is the 'Dying ReLU' problem in deep neural network optimization?
If a neuron's pre-activation drops into large negative values, ReLU yields an activation and gradient of 0, preventing future weight updates (the neuron 'dies').
Which activation function is designed to prevent neurons from becoming permanently inactive by providing a small non-zero slope alpha for negative inputs?
Leaky ReLU introduces a small positive slope (e.g., 0.01) for negative inputs, ensuring a non-zero gradient flows during backpropagation to prevent dying neurons.
Which activation function is used in the final output layer of a multi-class neural network to produce a normalized probability distribution across K mutually exclusive classes?
Softmax normalizes unconstrained real logit outputs into a valid probability distribution where all outputs lie in (0, 1) and sum to 1.0.