In decision tree algorithms, how is Shannon's Entropy H(S) mathematically defined for a dataset with classes having proportions p_i?
Shannon entropy quantifies impurity or information disorder: H(S) = - sum(p_i * log2(p_i)). When a node is completely pure, entropy is 0.
Which decision tree induction algorithm introduced by Ross Quinlan uses 'Information Gain' based on entropy as its splitting criterion?
Ross Quinlan introduced ID3 (Iterative Dichotomiser 3) using Information Gain. None of CART (uses Gini), Apriori (association rules), or K-Means (clustering) is Quinlan's algorithm.
Which splitting impurity metric is utilized by the CART (Classification and Regression Trees) algorithm for categorical classification trees?
CART uses Gini impurity, defined as 1 - $sum(p_i^2)$, to measure classification tree splits. It evaluates faster than entropy because it avoids logarithmic computation.
What bias is exhibited by standard Information Gain when evaluating splits on features with many distinct categorical values (such as Customer ID or Date)?
Information Gain favors high-cardinality attributes (like unique IDs) because splitting on them produces pure single-element leaf nodes with zero entropy, prompting C4.5 to use Gain Ratio instead.
In Support Vector Machines (SVM), what is the 'Optimal Separating Hyperplane'?
In SVM, the optimal separating hyperplane is defined as the decision boundary that maximizes the margin (geometric distance) between the closest points of the opposing classes (the support vectors).