Data Preparation
Exploratory analysis, cleaning, feature engineering and splitting — the work that sets your ceiling before any model is chosen.
Where the Work Actually Is
Your features set the ceiling. A good model on weak features loses to an ordinary model on strong ones, every time. Swapping algorithms moves a metric by a fraction of what a well-constructed feature does.
This section is also where the most damaging mistakes happen, because they are silent. A leak or a bad join does not throw an error — it produces an excellent validation score and a model that fails in production.
The order matters: look at the data, clean it, split it, then engineer features using only the training part. Doing those last two in the wrong order is the single most common serious error in applied machine learning.
Exploratory Analysis
Before any transformation, find out what you actually have. The goal is not pretty charts — it is discovering the problems early, while they are still cheap.
| Check | What you are looking for |
|---|---|
| Row and column count | Does it match what you were promised? |
| Missingness per column | Which columns are unusable, and is the missingness patterned? |
| Distribution of each numeric | Skew, bimodality, impossible values |
| Cardinality of each category | An ID masquerading as a feature |
| Duplicate rows | Usually a join that fanned out |
| Target balance | 1% positives changes everything downstream |
| Correlation with the target | Anything suspiciously perfect is a leak |
Plot before you summarise. Identical means and variances hide completely different shapes, and the shape decides which transformations make sense.
Cleaning & Missing Values
Ask why a value is missing before deciding what to do about it. The mechanism decides whether dropping is harmless or destroys your sample.
| Mechanism | Meaning | Dropping rows is… |
|---|---|---|
| MCAR | Missing completely at random | Safe, just wasteful |
| MAR | Missing depends on other observed columns | Biased unless you account for it |
| MNAR | Missing depends on the missing value itself | Actively dangerous |
Income left blank by high earners is MNAR: dropping those rows removes exactly the people you most need. Often the fact of missingness is itself predictive, so add a boolean was_missing column rather than pretending the gap never existed.
| Strategy | Use when | Cost |
|---|---|---|
| Drop the row | Few rows affected, MCAR | Loses data, can bias |
| Drop the column | Mostly empty and not important | Loses a signal you cannot recover |
| Mean or median fill | Numeric, small gaps | Shrinks variance, invents certainty |
| Category "Unknown" | Categorical | Honest, and often predictive |
| Model-based imputation | Gaps matter and data is rich | Complexity, and a new leak risk |
Outliers deserve the same question. A negative age is an error and should be fixed. A very large transaction may be the exact case you are trying to detect — deleting it because it is inconvenient is how fraud models get built that never fire.
Feature Engineering
Turning raw columns into things a model can use. Domain knowledge pays off here more than anywhere else in the pipeline.
| Technique | For | Notes |
|---|---|---|
| One-hot encoding | Low-cardinality categories | Explodes with many categories |
| Ordinal encoding | Categories with a real order | Implies spacing that may not exist |
| Target encoding | High-cardinality categories | Leaks badly unless fitted inside folds |
| Scaling | Distance and gradient models | Trees do not need it |
| Log transform | Right-skewed numerics | Prices, counts, durations |
| Binning | Non-linear effects | Discards information; use sparingly |
| Date parts | Timestamps | Day of week and hour usually beat the raw stamp |
Scaling matters for anything measuring distance or following a gradient — k-nearest neighbours, SVMs, neural networks, and regularised regression. Tree-based models split on thresholds and are indifferent to it, which is one reason gradient boosting is so forgiving on tabular data.
Train, Validation & Test Splits
Three sets, three jobs. Train fits the parameters. Validation chooses between models and settings. Test is looked at once, at the end, to estimate real performance.
Every time you tune against the test set you contaminate it a little, and its estimate drifts optimistic. That is what the validation set is protecting.
| Situation | Split it by | Because |
|---|---|---|
| Ordinary tabular data | Random | Rows are independent |
| Imbalanced target | Stratified | Keeps the rare class in every fold |
| Time series | Chronologically | You cannot train on the future |
| Several rows per user | By user (grouped) | Otherwise the same person is in both sides |
With limited data, k-fold cross-validation gives a more stable estimate than one split, because every row is used for validation exactly once. Keep a genuinely untouched test set anyway.
Common Mistakes
| Mistake | What it produces |
|---|---|
| Scaling or imputing before splitting | Leakage; an inflated score that never reproduces |
| Random split on time series | Training on the future; useless in production |
| Random split with repeat users | The same person on both sides of the split |
| Dropping every row with a gap | Silently biased sample when missingness is patterned |
| Deleting outliers by default | Removing the very events you were asked to detect |
| Target encoding outside folds | The strongest and most convincing leak there is |
| Tuning against the test set | No honest estimate of performance left |
Interview Questions
What is data leakage and how do you prevent it?
Any information available in training that would not exist at prediction time. Prevent it by splitting first and fitting every transformer — scalers, imputers, encoders — on the training split alone, then applying to the rest. Pipelines make this the default.
Why split time series chronologically?
A random split lets the model learn from the future to predict the past, which it can never do in production. Train on earlier periods, validate on later ones.
MCAR, MAR, MNAR — why does it matter?
It decides whether dropping rows is safe. MCAR is merely wasteful. MAR biases you unless you account for the related columns. MNAR — where missingness depends on the hidden value, like high earners omitting income — removes exactly the cases you need.
Which models need feature scaling?
Anything using distance or gradients: k-NN, SVMs, neural networks, regularised regression. Tree-based models split on thresholds and are unaffected.
Your validation accuracy is 99%. What do you check first?
Leakage. Look for a feature derived from the target, a row that appears on both sides of the split, or a transformer fitted before splitting. A near-perfect score on real-world data is a bug until proven otherwise.
Why three sets rather than two?
Validation is used repeatedly to choose models and hyperparameters, so it gradually stops being an honest estimate. The test set is held back and looked at once, at the end.