Train-Test Split and Cross-Validation
How to evaluate a model honestly: train, validation and test splits, k-fold cross-validation, and data leakage. A pure-noise experiment shows leakage producing 85% accuracy where the truth is 48%.
A train-test split holds data back to estimate how a model performs on new data, and k-fold cross-validation repeats that across k splits, testing each fold once and averaging the scores. Interviewers expect data leakage: any learned step, such as scaling or feature selection, must be fitted on training folds only, inside a Pipeline, or the score is overstated.
To estimate how a model will do on new data, hold some data back. A train-test split does it once, k-fold cross-validation repeats it across k different splits and averages the scores. Any step that learns from data, such as scaling or feature selection, must be fitted on training data only, or you leak information and overstate performance.
- The training set is for fitting, the validation set is for choosing models and settings, and the test set is used once at the end for the final estimate.
- In k-fold cross-validation, k models are trained, each holding out a different fold, so every row is used for testing exactly once and the k scores are averaged.
- 5-fold stratified cross-validation of a scaled logistic regression on the breast cancer dataset gave fold accuracies from 0.956 to 1.0, a mean of 0.979 and a standard deviation of 0.014.
- On pure random noise with 100 rows and 5,000 features, selecting features with all labels before cross-validation reported 85% accuracy, while selection inside the pipeline gave 48%.
- Time series data should never be shuffled for validation; forward-chaining splits such as TimeSeriesSplit always train on the past and test on the future.
01Why we hold data back
A model evaluated on its own training data will look good, and the more flexible it is, the better it will look. So we keep some data aside that the model never sees during training, and score it there. That score estimates how it will do in the real world.
The standard arrangement is three sets. The training set is for fitting. The validation set is for choosing between models and settings. The test set is touched once, at the end, for the final number. If you keep tuning against the test set, it turns into a second validation set and the final number becomes optimistic.
02The simple split
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)random_state makes the split repeatable. stratify=y keeps the share of each class the same in both parts, which matters when one class is rare.
The weakness of a single split is that the score depends on which rows landed in the test set. With a small dataset, a different split can change the result noticeably, and you have no idea how much.
03k-fold cross-validation
With 5-fold cross-validation, the data is cut into five parts. Five models are trained, each time holding out a different part as the test fold, so every row is used for testing exactly once. You get five scores, and their average and spread tell you more than one number.
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
scores = cross_val_score(pipe, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print(scores.round(3), scores.mean().round(3), scores.std().round(3))
# [0.956 0.974 0.982 1. 0.982] 0.979 0.014The five fold accuracies range from 0.956 to 1.0, with a mean of 0.979 and a standard deviation of 0.014. Reporting "97.9% plus or minus 1.4" is more honest than quoting one lucky split. Use stratified folds for classification, and larger k (10) for small datasets, at the cost of more training runs.
04Data leakage
Leakage is the most common reason a model works in a notebook and fails in production. Typical forms: scaling or imputing using the whole dataset before splitting, selecting features using the labels before splitting, duplicate rows ending up on both sides of a split, and a feature that is only known after the outcome (a "refund issued" column when predicting cancellations).
Here is how bad it can get. I made a dataset of pure random noise: 100 rows, 5,000 random features, random 0/1 labels. There is nothing to learn, so honest accuracy should be near 50%.
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, 100)
# Wrong: pick the 20 best features using ALL the labels, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
print(cross_val_score(LogisticRegression(max_iter=2000), X_leaky, y, cv=5).mean()) # 0.85
# Right: selection happens inside each training fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=2000))
print(cross_val_score(pipe, X, y, cv=5).mean()) # 0.48The leaky version reports 85% accuracy on pure noise. With 5,000 random features, some correlate with the labels by chance, and choosing them using the labels bakes the answer in. The correct version, with selection inside the pipeline so it only sees each training fold, scores 48%, which is the truth: no signal. The fix is simple: put every learned step in a Pipeline and let cross-validation fit it per fold.
05Special cases
Time series: never shuffle. Train on the past and test on the future, like TimeSeriesSplit, or you let the model peek ahead.
Grouped data: if one patient or user has many rows, keep all of a group's rows on the same side with GroupKFold, otherwise the model recognises the person rather than learning the pattern.
Very small data: use repeated or leave-one-out cross-validation, accepting more compute for a steadier estimate.
After you have chosen a model with cross-validation, retrain it on all the training data. Keep a final untouched test set, if you can afford one, for the last check.
06Common interview questions
Q: Why not evaluate on the training set?
A: It rewards memorisation and says nothing about unseen data.
Q: What is the validation set for?
A: Choosing models and hyperparameters, so the test set stays untouched for an unbiased final estimate.
Q: How does k-fold CV work?
A: Split into k folds, train on k-1, test on the held-out fold, repeat so each fold is the test once, then average the scores.
Q: What is data leakage? Give an example.
A: Test or future information influencing training, for example fitting a scaler on the full dataset before splitting.
Q: How do you cross-validate time series data?
A: With forward-chaining splits where training data always precedes test data.
Q: When use stratified splits?
A: For classification, especially with imbalanced classes, to keep class proportions in each fold.