What you will take away
- A model is a deployed dependency on data that keeps changing.
- Version data and features, not just code and weights.
- Monitor inputs as closely as outputs - drift shows up there first.
Why models decay and software does not
A deployed function keeps behaving identically until someone changes it. A deployed model degrades on its own, because the world it was trained on moves: customer behaviour shifts, a competitor changes pricing, an upstream system starts sending a field in a new format.
That single difference is the reason machine learning needs operational practice beyond ordinary deployment. The question is never only "did it ship" but "is it still right this week".
Reproducibility first
Before anything else, you must be able to rebuild a model exactly: the code, the data snapshot, the feature definitions, the hyperparameters and the environment. Without that, you cannot debug a production regression, because you cannot reconstruct what was actually deployed.
- Version training data by snapshot or by a query pinned to a timestamp.
- Keep feature definitions in one place used by both training and serving.
- Record every experiment with its parameters and metrics, automatically.
- Register each candidate model with its lineage, evaluation and approval state.
Training and serving computing a feature slightly differently is the most common production bug in machine learning, and the hardest to see.
Monitoring the inputs, not only the outputs
Accuracy is often unknowable in real time, because the ground truth arrives weeks later. Input monitoring is available immediately: distribution shift per feature, null rates, unexpected categories, volume anomalies. Most incidents announce themselves there long before the business metric moves.
Alert on the inputs, and keep every prediction with its features so that when the outcomes do arrive, accuracy can be computed retrospectively against exactly what the model saw.
Rolling out a new model safely
Never swap a model in one step. Run the candidate in shadow mode against live traffic first, comparing predictions without acting on them. Then move a small percentage of decisions, watch the business metric rather than the offline score, and expand.
Keep the previous version deployable and the switch immediate. Rolling back a model must be as fast and as boring as rolling back an application release.
Retraining on a trigger, not a calendar
Monthly retraining is a habit, not a strategy. Retrain when the monitoring says something has changed: drift beyond a threshold, a measurable drop once labels land, or a known business event such as a pricing change or a new market.
Automate the pipeline so retraining is a routine run rather than a project, and keep a human approval step for the promotion decision. The automation should make the retrain cheap; it should not make the deployment unattended.
OrbixEdge Engineering
The engineers who build and run the systems we write about - platform, cloud and product teams shipping for clients in retail, health, logistics and fintech.
