A fully connected neural network has 4 input features, a hidden layer with 5 neurons, and an output layer with 2 neurons. Including bias terms for all hidden and output neurons, what is the TOTAL number of trainable parameters in this network?
Input -> Hidden: (4 * 5) weights + 5 biases = 25. Hidden -> Output: (5 * 2) weights + 2 biases = 12. Total = 25 + 12 = 37.
In Inverted Dropout training with dropout probability p = 0.20, active neuron outputs are multiplied by what scaling factor during forward pass?
Scaling factor = 1 / (1 - p) = 1 / (1 - 0.20) = 1 / 0.80 = 1.25.
A single neuron with Sigmoid activation $\sigma(z) receives input z = 0$. What is the value of the derivative $\sigma'(0)$?
$\sigma(0) = 0.5. Derivative \sigma'(0) = \sigma(0) * (1 - \sigma(0)) = 0.5 * 0.5 = 0.25.$
A fully connected layer takes an input vector of dimension 100 and produces an output vector of dimension 50. How many WEIGHT parameters (excluding biases) are in this layer?
Weight matrix size = 50 x 100 = 5000 weights.
If a neural network trained with Cross-Entropy loss predicts probability p = 0.5 for a target class with ground-truth label y = 1, what is the Cross-Entropy loss for this instance? (Use natural log, round to 2 decimal places)
$Loss = - \ln(0.5) = 0.6931 ~ 0.69.$