When building a Machine Learning model, we usually do not use the entire dataset for training. Instead, the available data is divided into different parts so that we can train the model, tune it, and evaluate it fairly. The three important parts are:
- Training Set
- Validation Set
- Test Set
Machine Learning Data Splitting #
1. Training Set #
Simple Explanation #
Example #
Suppose we have a dataset containing information about 10,000 students. We may use:
8,000 students → Training Set
The model learns the relationship between:
- IQ
- CGPA
- Study Hours
- Internship Experience
and the target: Placement.
2. Validation Set #
Simple Explanation #
Why Do We Need Validation Data? #
Suppose we are using a KNN model. We need to decide the value of K. For example:
- K = 3
- K = 5
- K = 7
- K = 9
We can use validation data to compare these choices and select a suitable configuration.
Suppose we train a model with different learning rates:
| Learning Rate | Validation Accuracy |
|---|---|
| 0.1 | 84% |
| 0.01 | 91% |
| 0.001 | 87% |
The validation results help us decide which configuration works better during model development.
3. Test Set #
Simple Explanation #
Example #
Suppose a dataset contains 10,000 samples. After training and validation, we keep a separate test set.
Test Set = 1,000 samples
After the model and its configuration have been finalized, we evaluate it on these 1,000 unseen samples.
4. Why Is Dataset Splitting Required? #
Simple Explanation #
If we train and evaluate a model on exactly the same data, the evaluation may not represent how the model performs on new, unseen data. A model could simply become very good at the examples it has already seen. Therefore, we separate the data.
Main Reasons for Splitting Data #
- To train the model using one portion of the data.
- To tune hyperparameters and compare models.
- To evaluate the final model on unseen data.
- To estimate how well the model may generalize to new data.
- To reduce the risk of evaluating the model only on data it has already seen.
5. Common Train / Validation / Test Ratios #
There is no single train-validation-test ratio that is correct for every Machine Learning problem. The appropriate split depends on factors such as dataset size, problem type, and available data.
Common Examples #
| Training | Validation | Testing |
|---|---|---|
| 70% | 15% | 15% |
| 80% | 10% | 10% |
| 80% | 20% | — |
| 90% | 5% | 5% |
Using an 80 / 10 / 10 split:
Training = 8,000 samples
Validation = 1,000 samples
Testing = 1,000 samples
6. Data Leakage #
Simple Explanation #
Simple Example #
Suppose you have:
- Training Data
- Validation Data
- Test Data
You repeatedly check the test-set performance while changing your model and hyperparameters.
Eventually, your decisions start being influenced by the test set.
Now the test set is no longer acting as a clean final evaluation set.
Another Important Example: Preprocessing Leakage #
Suppose you want to standardize your entire dataset. A common mistake is to calculate the mean and standard deviation using the complete dataset before splitting it. This allows information from validation/test samples to influence the transformation used for training.
A safer approach is:
7. Training vs Validation vs Test Set #
| Set | Main Purpose | Used During | Easy Word |
|---|---|---|---|
| Training | Learn model parameters and patterns | Training | Learn |
| Validation | Tune hyperparameters and compare model configurations | Model development | Tune |
| Test | Final evaluation on unseen data | After model decisions are finalized | Evaluate |
Training → Learn
Validation → Tune
Testing → Final Check
8. Complete Train-Validation-Test Workflow #
9. Real-Life Example #
Imagine Preparing for an Exam #
Imagine you have 1,000 practice questions.
Training Set: You study and practice 800 questions.
Validation Set: You use 100 questions to check your preparation and decide which topics or study strategies need improvement.
Test Set: You finally attempt 100 questions that you have kept aside for the final assessment.
The idea is similar in Machine Learning:
- Training → Learn
- Validation → Improve / Tune
- Testing → Final evaluation
Training Set = Study Material / Practice
Validation Set = Mock Test for Improvement
Test Set = Final Exam
10. Quick Revision for Exams #
Training Set: Used to learn model parameters and patterns.
Validation Set: Used for model selection and hyperparameter tuning.
Test Set: Used for final evaluation on unseen data.
Why Split Data? To train, tune, and evaluate the model separately.
Common Ratios: Examples include 80/20 for train/test and 70/15/15 or 80/10/10 for train/validation/test.
Data Leakage: Occurs when inappropriate information from outside the training process influences model learning or evaluation.
11. One Diagram to Remember Everything #
Conclusion #
Splitting data is one of the fundamental steps in a Machine Learning workflow. The key idea is:
Validation → Tune
Testing → Final Evaluation
A good Machine Learning workflow keeps the test set separate from model-development decisions and takes care to avoid data leakage.
Training, Validation & Test Data in Machine Learning Quiz #
1. What is the main purpose of the training set in Machine Learning?
To perform the final evaluation
To learn patterns and model parameters from data
To select the test set
To calculate only the accuracy
Explanation
The training set is used to train the Machine Learning model by learning patterns and model parameters from the available data.
2. What is the main purpose of a validation set?
To train the model parameters only
To permanently store predictions
To tune hyperparameters and compare model configurations
To replace the training set
Explanation
The validation set is used for hyperparameter tuning, model selection, and comparing different model configurations during development.
3. What is the primary purpose of the test set?
To repeatedly tune hyperparameters
To train model parameters
To perform the final evaluation on unseen data
To increase the training dataset
Explanation
The test set is used for the final evaluation of a trained and finalized model on data that was not used during model development.
4. Why is a dataset split into training, validation, and test sets?
To make the dataset smaller
To train, tune, and evaluate the model separately
To remove all features
To avoid using Machine Learning algorithms
Explanation
Dataset splitting allows the model to learn from training data, tune decisions using validation data, and receive a final evaluation using unseen test data.
5. Which dataset is primarily used to learn the parameters of a Machine Learning model?
Training set
Validation set
Test set
Production set
Explanation
The training set is used to learn the model parameters and patterns from the data.
6. Which of the following is a common train/test split when a separate validation set is not used?
50/50
60/40
80/20
99/1
Explanation
An 80/20 split is a common example where 80% of the data is used for training and 20% for testing. The appropriate ratio depends on the problem and dataset.
7. Which is an example of a three-way train/validation/test split?
70/15/15
30/30/40
50/10/40
95/10/5
Explanation
70/15/15 is a valid three-way split: 70% training, 15% validation, and 15% testing. Other splits such as 80/10/10 can also be used.
8. A dataset contains 10,000 samples. Using an 80/10/10 split, how many samples are used for testing?
100
500
1,000
8,000
Explanation
With an 80/10/10 split, 10% of 10,000 samples are assigned to the test set, which equals 1,000 samples.
9. What does generalization mean in Machine Learning?
The ability to memorize the training data
The ability to perform well on new, unseen data
The ability to increase the dataset size
The ability to remove validation data
Explanation
Generalization is the ability of a Machine Learning model to perform well on new, unseen data rather than only on its training examples.
10. What is data leakage?
Removing duplicate rows from a dataset
Using inappropriate information during training or evaluation that should not be available
Splitting data into training and testing sets
Reducing the number of features
Explanation
Data leakage occurs when information from outside the appropriate training process becomes available to the model during training or development, often producing overly optimistic evaluation results.
11. Which situation is an example of data leakage?
Training a model using only the training set
Evaluating the finalized model once on the test set
Repeatedly changing the model based on test-set performance
Splitting the dataset before training
Explanation
Repeatedly using test-set results to change the model or hyperparameters causes information from the test set to influence development, which compromises its role as an unbiased final evaluation.
12. What is a common form of preprocessing data leakage?
Fitting a scaler only on the training data
Calculating preprocessing statistics using the complete dataset before splitting
Applying the trained scaler to test data
Removing missing values from training data
Explanation
Calculating preprocessing statistics such as mean and standard deviation using the complete dataset before splitting can allow information from validation or test data to influence the training process.
13. How should a scaler generally be used to avoid preprocessing leakage?
Fit it separately on the test set
Fit it on the complete dataset
Fit it on the training data and apply the learned transformation to validation/test data
Fit it only on the target variable
Explanation
The scaler should be fitted using the training data, and the same learned transformation should then be applied to validation and test data.
14. Which dataset should generally remain untouched while selecting hyperparameters?
Training set
Validation set
Test set
All datasets equally
Explanation
The test set should generally remain untouched during model selection and hyperparameter tuning so it can provide a more unbiased final evaluation.
15. What can happen if the test set is used repeatedly for hyperparameter tuning?
The test set becomes completely independent
The model may indirectly overfit to the test set
The training set becomes larger
Data leakage becomes impossible
Explanation
Repeatedly using test-set results for tuning can cause the model-development process to overfit to the test set, making the final evaluation less reliable.
