GATE DA Supervised Learning – Data Science & AI | Set 1

Original solution basis:
Continuous features in Gaussian Naive Bayes are modeled as 1D Gaussians P(x_i | c) = (1 / sqrt(2 pi sigma_c^2)) $\exp(-(x_i - mu_c)^2 / (2 sigma_c^2))$.

Deep Dive
Gaussian Naive Bayes models each continuous feature, conditional on a class c, using a normal density with class-specific mean \mu _c and variance $\sigma _c²$. The density is [1/√(2$\pi \sigma _c²)] exp[-(x-\mu _c)²/(2\sigma _c²)]$. Parameters are estimated separately for each feature and class; conditional independence is then used to combine the densities.

Key Points to Remember:
• Gaussian NB uses a univariate Gaussian likelihood per feature and class.
• Each class-feature pair has its own mean and variance.
• The class posterior combines prior probability with the product of feature likelihoods.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
By Bayes' rule, MAP drops the constant marginal denominator P(X): argmax_c P(c | X) = argmax_c P(c) prod_i P(X_i | c).

Deep Dive
Bayes' theorem gives P(c|X) ∝ P(c)P(X|c). Under the Naive Bayes conditional-independence assumption, P(X|c)=ΠᵢP(Xᵢ|c). The denominator P(X) is identical for every candidate class, so it can be omitted when taking argmax. The predicted class is the one with the largest prior-times-likelihood product.

Key Points to Remember:
• MAP decision: argmax_c P(c)ΠᵢP(xᵢ|c).
• The evidence P(X) is a class-independent normalization constant.
• In practice, log probabilities are summed to avoid numerical underflow.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Entropy measures impurity: H(S) = - sum p_i log2(p_i). For binary classes: -p+ log2(p+) - p- log2(p-).

Deep Dive
For class probabilities p₊ and p₋, entropy is H(S)=-p₊log₂p₊-p₋log₂p₋. The negative signs ensure entropy is nonnegative because log₂(p)$\le 0$ for probabilities in (0,1]. Entropy is zero for a pure node and reaches one bit when the two classes are equally likely.

Key Points to Remember:
• Entropy: H=-$\Sigma ᵢpᵢlog₂pᵢ$.
• Use the convention 0 log₂0 = 0.
• Binary entropy is maximal at p₊=p₋=0.5, where H=1 bit.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Information Gain is parent entropy minus expected weighted child entropy: H(S) - sum_v (|$S_v$|/|S|) H(S_v).

Deep Dive
Information Gain compares the parent entropy with the weighted average entropy after splitting. Each child entropy is weighted by its fraction |$S_v$|/|S| of the parent samples. Therefore IG(S,A)=H(S)-$\Sigma _v(|S_v|/|S|)H(S_v)$. A useful split has low weighted child impurity and hence high information gain.

Key Points to Remember:
• Information Gain = parent entropy - weighted child entropy.
• Child nodes must be weighted by their sample proportions.
• A split that creates pure children has maximum possible gain for that parent.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Gini impurity measures probability of misclassifying a randomly chosen element: Gini = 1 - sum $p_i^2$.

Deep Dive
For K classes with probabilities p₁,...,$p_K$, Gini impurity is 1-$\Sigma ᵢpᵢ²$. The quantity \Sigma pᵢ² is the probability that two independently drawn labels belong to the same class; subtracting from one gives the probability of different classes. A pure node has Gini 0.

Key Points to Remember:
• Gini = 1-$\Sigma pᵢ²$.
• Gini is 0 for a pure node.
• For two equally represented classes, Gini is 0.5.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.