GATE DA Supervised Learning – Data Science & AI | Set 1

Original solution basis:
Stratified CV maintains class distribution proportions across folds, essential for imbalanced classification tasks.

Deep Dive
Stratified K-fold performs the same rotation of training and validation folds but preserves approximately the overall class proportions in each fold. This is especially useful when one class is rare, because an ordinary random split might produce folds with very few or no minority-class examples.

Key Points to Remember:
• Stratification preserves approximate class proportions in every fold.
• It is particularly useful for imbalanced classification.
• It does not guarantee identical counts when class counts are not divisible by K.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Bagging averages B trees trained on bootstrap samples. Averaging reduces prediction variance without increasing bias.

Deep Dive
Bagging draws multiple bootstrap samples from the training set and fits a base learner to each sample. Predictions are combined by voting or averaging. Because the learners see different resampled datasets, averaging can reduce variance, especially for unstable learners such as deep decision trees. Bagging does not primarily work by sequentially correcting residuals.

Key Points to Remember:
• Bagging = bootstrap sampling + parallel base learners + aggregation.
• Its main benefit is variance reduction.
• Random Forest is a bagging-based method with additional feature subsampling.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Bagged trees are correlated if a strong feature dominates root splits. Random Forest selects m ~ sqrt(d) random features per split to decorrelate trees.

Deep Dive
A standard bagged tree considers all available features at each split. Random Forest randomly restricts the candidate feature set at each node. This reduces correlation among trees; averaging less-correlated trees can lower ensemble variance and improve generalization.

Key Points to Remember:
• Random Forest combines bootstrap sampling with random feature selection at splits.
• Feature subsampling decorrelates trees.
• The number of candidate features is a tunable hyperparameter.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
Boosting is a sequential ensemble method that converts high-bias weak learners into a strong learner by fitting residuals/reweighting misclassified samples.

Deep Dive
Boosting builds learners sequentially. Each new learner is fitted to address weaknesses left by the current ensemble: AdaBoost increases emphasis on misclassified examples, while gradient boosting fits negative gradients or pseudo-residuals. The weighted/additive combination of weak learners forms a stronger predictor.

Key Points to Remember:
• Boosting is sequential, unlike the usual parallel bagging setup.
• AdaBoost reweights difficult examples; gradient boosting fits pseudo-residuals.
• Learning rate and number/depth of learners control the strength of boosting.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.

Original solution basis:
AdaBoost updates weight w_i <- w_i * \exp(alpha_m) for misclassified instances, forcing the next weak learner to focus on hard samples.

Deep Dive
In AdaBoost, the weak learner receives a coefficient $\alpha ₘ based on its weighted error$. Before normalization, an example misclassified by that learner has its weight multiplied by $\exp(\alpha ₘ), increasing its relative importance$. Correctly classified examples are multiplied by $\exp(-\alpha ₘ), decreasing their relative importance$. The weights are then normalized to sum to one.

Key Points to Remember:
• Misclassified examples receive a relative weight increase by $\exp(\alpha ₘ)$.
• Correctly classified examples receive a relative decrease by $\exp(-\alpha ₘ)$.
• Weights are normalized after each update.

Exam tip:
Identify the governing definition or theorem first, write the formula, substitute the given values, and check dimensions/signs before selecting the option.