What you will take away
- Start from a decision somebody makes weekly, not from a dataset.
- A naive baseline is the only honest way to judge a model.
- A prediction nobody acts on has zero value regardless of accuracy.
Start from the decision, not the data
The projects that fail begin with a data warehouse and a hope. The ones that pay for themselves begin with a specific decision someone makes on a schedule: how much to order, which customers to call this week, what to charge for the last twenty seats.
Naming the decision gives you the accuracy bar, the deadline, the format and the owner. It also tells you when to stop: a model that beats the current process by enough to change the decision is finished, however unimpressive the score looks in isolation.
The three that reliably return their cost
- Demand forecasting, where a few percentage points of accuracy translate directly into stock that is not sitting in a warehouse.
- Churn prediction, which is only useful with a retention action attached - a scored list with no play behind it changes nothing.
- Price and discount optimisation, where the constraint is usually policy and margin rules rather than the model.
What these share is a short feedback loop. The prediction is checked against reality within weeks, so the model can be retrained on its own errors and the business can see whether the investment worked.
Data you need, and data you think you need
Two years of clean transactional history usually beats five years of history with an undocumented system change in the middle of it. Before modelling, establish what changed in the business and when - a pricing change, a new channel, a supply disruption - because those events are the reason a model trained on the past mispredicts the present.
Also watch for leakage, which is the most common reason a model looks excellent in testing and fails in production. If a feature is only populated after the outcome is known, the model has been shown the answer.
A model that scores ninety-five per cent in a notebook and fifty in production has almost always been given a field it will not have on the day.
Always build the naive baseline
Before any machine learning, implement the obvious rule: last week repeated, a seasonal average, the current manual heuristic. Measure it on the same period and metric. Surprisingly often it is within a few per cent of the sophisticated model, which is an extremely useful thing to learn in week one rather than month six.
When the model does beat the baseline, the baseline is what you use to express the value in business terms - not as an accuracy figure, but as units of stock, hours or margin.
Delivery is the project
The forecast must arrive where the decision is made: inside the purchasing screen, in the ordering system, in the weekly list the account team already works through. A separate dashboard that requires someone to remember to open it will be checked enthusiastically for three weeks and then not at all.
Show the uncertainty alongside the number, log every prediction with the inputs that produced it, and monitor drift. A model is not a deliverable that is handed over. It is a system that has to be watched, and budgeted for accordingly.
OrbixEdge Engineering
The engineers who build and run the systems we write about - platform, cloud and product teams shipping for clients in retail, health, logistics and fintech.
