MLOps best practices that hold up after launch come down to five habits. Version data, code and model together so any release can be rebuilt. Test the whole pipeline, not just model accuracy. Gate each release on a comparison with the current model plus a named approver. Monitor drift, quality and cost. Keep the last good model one switch away.
Most teams that struggle with production ML don't have a modelling problem. The model scored well in a notebook. Then the input data changed, a feature was computed differently at serving time, or nobody could say which dataset produced the model that's now misbehaving. The practices below are aimed at those failures.
What does MLOps add that standard DevOps does not cover?#
MLOps treats data and model behaviour as things that change and need testing, because a model can get worse without anyone touching the code.
Google's architecture guidance defines MLOps as an engineering culture and practice that brings ML system development and ML system operation together. It adds a third loop to CI and CD: continuous training (CT). The same document says the real challenge isn't building a model. It's building an integrated ML system and running it continuously in production.
The cost of skipping this is well documented. Google engineers' NeurIPS paper on hidden technical debt in ML systems found that large ongoing maintenance costs are common in real-world ML systems. It warns that one or two harmless-looking data dependencies can slow later progress.
| Concern | Standard DevOps covers | MLOps adds |
|---|---|---|
| What changes | Code and config | Code, config, training data, features, model weights |
| What gets tested | Behaviour against a spec | Data schemas, distributions, model quality on data slices |
| Why it breaks | A bad deploy | A bad deploy, or the world changing under a fixed model |
| What gets released | A build artifact | A model artifact plus the lineage that produced it |
| Monitoring | Latency, errors, saturation | Those, plus input drift, prediction drift and outcome quality |
How do you version data, code and models so any release can be reproduced?#
Treat each model release as one record in a model registry that links the code commit, data snapshot, feature definitions, training config, environment and the model artifact.
If any of those six is missing, you can't reproduce the release, and debugging turns into guesswork. In practice:
- Code: the training and serving code at a specific commit, reviewed like any other change.
- Data: an immutable snapshot or table version, not "the training bucket as of last Tuesday". Open table formats with time travel make this cheap.
- Features: transformation logic in version control, so training and serving compute a feature the same way.
- Config: hyperparameters, random seeds and evaluation thresholds stored as files, not notebook cells.
- Environment: a pinned container image, including library versions.
- Model artifact: the trained weights, registered with a version number and links to all of the above.
The data side is usually the weakest. A data platform where transformations live in version control and run through CI gives the model the same footing as the application code. So does feeding feature stores and vector indexes from the same governed tables.
A simple test of whether your versioning works: pick the model serving traffic today and ask an engineer to rebuild it from the registry record alone. If they need to ask someone, there's a gap.
What should a model pipeline test before a model reaches production?#
Test four layers: the incoming data, the training code, the trained model and the serving path. Model accuracy is only one of them.
Google's ML Test Score rubric sets out 28 specific tests and monitoring needs drawn from its production ML systems. It's a useful checklist for scoring your own readiness. The core of it maps to these layers:
- Data tests: schema checks (types, ranges, allowed values), null and duplicate rates, and distribution checks against the last training set. Fail the run if a required feature disappears or its range shifts outside agreed bounds.
- Training code tests: unit tests for feature transforms, plus a quick training run on a small fixed sample to confirm that loss goes down and the run is deterministic.
- Model tests: quality on the full holdout set and on named slices (regions, customer tiers, rare classes). A higher overall score can hide a drop on one slice.
- Serving tests: send the same inputs through the training transforms and the serving path and compare the features. This catches training-serving skew, a common cause of "great offline, poor online". Add latency and memory checks at expected load.
For generative AI features, the model test is an evaluation set. On our generative AI work, we write it from real user questions before choosing a model or prompt. Those scored questions then run again on every prompt, model or retrieval change.
Pipeline code is also code, and more of it is now written by AI assistants. It should pass the same tests, security checks and senior review as any other change. Our guide to verifying AI-generated code before production covers those gates.
How do you set release gates and approvals for a model?#
A model ships only when it matches or beats the production model on the same evaluation data and clears agreed thresholds. A named owner signs off, and it then reaches users through shadow and canary traffic.
Write the gate criteria down before training starts, so nobody negotiates thresholds after seeing the results. A workable gate includes:
- Comparison, not an absolute score: the candidate is evaluated on the same frozen evaluation set as the current production model.
- Slice floors: no named slice may get worse by more than an agreed margin, even if the overall score improves.
- Operational limits: p95 latency, memory and cost per prediction within budget.
- Safety and policy checks: for LLM features, guardrail and refusal tests. For regulated decisions, fairness checks on the agreed groups.
- Human approval: a model owner reviews the evaluation report and approves or rejects in the registry, which records who decided and when.
Then roll out in stages. In shadow mode, the candidate scores live traffic without affecting users. In canary mode, it serves a small share of traffic while you compare it with production.
Keep these records even if no regulator asks for them yet. The EU AI Act's Article 12 requires high-risk AI systems to allow automatic logging of events over the system's lifetime, including for post-market monitoring. For Annex III high-risk systems this applies from 2 December 2027. A registry that already logs lineage, evaluation results and approvals gives you most of that evidence.
How do you monitor a model for drift, quality and cost after launch?#
Monitor four signals, each with an owner and a threshold: input drift, prediction drift, outcome quality once labels arrive, and serving health and cost.
Input drift. Compare live feature distributions with the training reference. The two-sample Kolmogorov-Smirnov test checks whether two samples come from the same distribution, without assuming its shape. The Population Stability Index has a common rule of thumb: below 0.1 is stable, 0.1 to 0.2 is moderate change, and above 0.2 is a significant shift. Tune these thresholds on your own history instead of copying them.
Prediction drift. Track the distribution of model outputs. A fraud model that suddenly flags twice as many transactions is telling you something, even before you have labels.
Outcome quality. Labels often arrive late: chargebacks take weeks, churn takes months. Track the real metric when it arrives, and use proxies in the meantime, such as overrides by human reviewers or user corrections.
Serving health and cost. Watch latency, error rates and cost per prediction. For LLM features, set latency and cost budgets per feature and use caching and token limits to keep the cost of each request predictable.
Drift on its own isn't an incident. Here's how to read an alert: if inputs shifted but quality held, look at the upstream data before touching the model. If quality dropped, first check whether a pipeline bug caused it, such as a broken join or a renamed field. If so, fix the data and replay. If not, roll back or retrain.
These signals belong in the same alerting and on-call setup as the rest of production. Model alerts that go to a data scientist's inbox get read days late. Observability and SRE practices apply directly: set SLOs that users would recognise, instrument with OpenTelemetry and cut alerts down to the ones that matter.
What is a safe rollback plan when a model starts to fail?#
Keep the previous production model warm as a standby, route traffic with a flag or weight you can change in seconds, and agree the rollback triggers before launch. That way, rolling back is a decision, not a debate.
NIST's generative AI profile of the AI Risk Management Framework asks that a deployed system be shown to be safe and able to fail safely. For a model, failing safely needs four things:
- A standby model. When a new model is promoted, the old one stays deployed or one config change away. It isn't deleted.
- Traffic control. A feature flag or weighted routing sends traffic back without a redeploy.
- Pre-agreed triggers. For example: canary error rate above budget, a quality proxy down more than an agreed margin, or PSI past the alert line together with a drop in quality.
- A last-resort fallback. A rules-based default, a cached answer or a "needs human review" path for when no model is trustworthy. For LLM features, route across providers when one slows down or fails.
Remember that the fault may sit in the data, not the model. If a feature pipeline broke, rolling back the model won't help. You need to fix the data and replay it, which only works if the data is versioned.
Archived versions keep their registry record and artifacts, so you can still rebuild them and explain what they did. Rehearse rollback once per quarter on a non-critical model. A rollback you have never practised may not work when you need it.
Conclusion#
A model that works in a notebook becomes a production system once it's versioned with its data, tested at every layer, released through a gate with a named approver, monitored for drift and cost, and backed by a standby model. None of this calls for a new platform on day one. It needs the same discipline as good software delivery, applied to data and models as well as code.
If you'd like help putting these practices around one production model, start with a paid pilot: one pod, one outcome, success criteria agreed up front. Talk to an engineering lead →
Frequently asked questions
Do LLM applications need MLOps, or something different?
Most of the same practices apply, but what you version and test changes. Prompts, retrieval settings and model choices need versioning like code, and an evaluation set of real user questions replaces the classic holdout set. Cost per request and latency budgets also become first-class release criteria.
How often should a production model be retrained?
Base it on triggers, not the calendar. Retrain when monitoring shows sustained drift together with a drop in outcome quality, or when new labelled data covers cases the model handles poorly. A scheduled retrain can still be useful, but it should go through the same evaluation gate as any other candidate.
What is the minimum MLOps setup for a first production model?
A registry entry that links the code commit, data snapshot and model artifact. Add data schema checks in the pipeline, an evaluation against a frozen test set, basic drift and latency monitoring, and a previous version you can switch back to. You can automate further once these basics work.
Who should own a model once it is in production?
One named owner should be accountable for its quality, approve releases and respond to its alerts, just as a service owner does for an application. Data scientists can build the model, but the alerts need to reach whoever is on call for that product area. Otherwise drift gets noticed days late.
- MLOps
- AI in production
- Machine learning
- Model monitoring
- Model governance
Newsletter
Get new articles by email
A short monthly round-up of what we've written on engineering pods, quality and AI. No spam.
