MLOps
Experiment tracking, model serving, and noticing when the data moves under you — the work between a notebook and a product.
A Notebook Is Not a Product
Machine learning systems fail in ways ordinary software does not. Normal code breaks loudly. A model degrades silently: it keeps returning confident answers while the world it was trained on drifts away underneath it.
There are also three moving parts to version, not one. Code, data and model must be reproducible together — the same code on different data gives a different model, and knowing which combination produced the artefact currently serving traffic is a basic requirement.
Most of this discipline is ordinary engineering applied to an unusual artefact: version control, CI, containers, monitoring, rollback. If you know how to run a service, you are most of the way there.
Experiment Tracking
Training runs are experiments, and experiments need records. Without one you cannot answer the question that always arrives: which run produced the model we are serving, and can we reproduce it?
| Record per run | Why |
|---|---|
| Code version | The commit, so the run can be recreated |
| Data version | Same code, different data, different model |
| Hyperparameters | Otherwise the best run cannot be repeated |
| Metrics | Comparison across runs, not just the last one |
| Environment | Library versions change results more than people expect |
| The artefact | The weights themselves, addressable |
A model registry is the next step up: named, versioned models with a stage — staging, production, archived — so promotion and rollback are explicit acts rather than someone copying a file.
Model Serving
How predictions reach the thing that needs them. The choice is mostly about latency tolerance.
| Pattern | Shape | Fits |
|---|---|---|
| Batch | Score everything nightly, write to a table | Churn scores, recommendations refreshed daily |
| Online | An API answering one request at a time | Fraud checks, search ranking, anything interactive |
| Streaming | Score events as they arrive | Live anomaly detection |
| Embedded | The model ships inside the client | On-device, offline, privacy-sensitive |
An online model is a web service with an unusual payload, so everything you know applies: containerise it, put it behind a load balancer, set timeouts, and have a fallback for when it is slow — often the previous model, or a simple heuristic.
Ship a new model the way you ship anything risky: to a fraction of traffic first, compared against the incumbent, with a rollback that has actually been rehearsed.
Monitoring & Drift
The distinguishing problem of running models: performance decays without anything breaking. Nothing throws, no alert fires, and the metric that matters may not be measurable for weeks.
| What moved | Name | Example |
|---|---|---|
| The inputs | Data drift | A new market shifts the age distribution |
| The relationship | Concept drift | Fraud tactics change, so the same signals stop meaning fraud |
| The pipeline | Schema skew | An upstream column starts arriving null |
Because ground truth is delayed — you learn whether a loan defaults months later — monitor the inputs and the outputs rather than waiting for accuracy. A shift in the distribution of predictions is an early warning you can act on today.
| Watch | Tells you |
|---|---|
| Feature distributions vs training | Data drift, before accuracy moves |
| Prediction distribution | The model's behaviour changing |
| Null and default rates | An upstream pipeline breaking |
| Latency and error rate | It is still a service |
| Accuracy once labels land | The truth, late |
Interview Questions
Data drift versus concept drift?
Data drift is the input distribution moving — a new market, a new device mix. Concept drift is the relationship between inputs and target changing, so the same inputs now mean something different. Data drift can be harmless; concept drift always degrades the model.
What is training/serving skew?
Features computed differently at training and serving time — a different default, unit, or time window. The model receives inputs it was never trained on and accuracy drops with no error anywhere. The fix is sharing one feature implementation across both paths.
Ground truth arrives weeks late. What do you monitor?
Inputs and outputs. Feature distributions against the training baseline, the distribution of predictions, and pipeline health — nulls, defaults, volumes. Those move before accuracy does, and they are measurable immediately.
Batch or online serving?
Latency decides. If predictions can be a day old, batch is cheaper, simpler and easier to backfill. If the answer must reflect this request, you need online, and with it the availability and latency obligations of any service.
What has to be versioned to reproduce a model?
Code, data and hyperparameters, plus the environment and the random seed. Code alone is not enough — the same code on different data is a different model.
How would you deploy a new model safely?
Register the version, deploy to a fraction of traffic, compare against the incumbent on live data, and keep a rehearsed rollback. Treat it as a risky release, because a model that scores better offline can still be worse in production.