The models were beautiful in the notebook and brittle in the world. Each new demand forecast looked like a step change in accuracy right up until the week it shipped, when a holiday, a promotion, or a weather event it had never seen would send it sideways — and the ops team would quietly go back to their spreadsheet.
The problem, precisely
The failure wasn’t the model; it was the absence of everything around the model. No honest backtest, no drift detection, no way for the people who actually run operations to say “not this week.” A forecast the team can’t override is a forecast the team learns to ignore.
The approach
We started by building the least glamorous thing in the project: a boring baseline nobody was allowed to beat without proving it on a real backtest. Then drift monitoring to catch the model wandering, and an override workflow so ops stayed in command.
A model your operators can veto is a model your operators will actually use.
What we built
# forecasting loop
baseline → deliberately boring, hard to beat
backtest → honest replay on unseen periods
serve → forecast + confidence band
monitor → drift + error alerts, not silent decay
override → ops can veto, with the reason logged
How we knew it worked
The baseline held for twelve weeks before any model earned its way past it — which was the point. When the learned model finally shipped, forecast error dropped 22%, on-call pages fell 60%, and every forecast stayed overridable by the ops team, with each veto logged as training signal for the next round.