It is entirely possible to build a more accurate forecast and see no meaningful improvement in inventory, service level or capacity. The explanation is uncomfortable because it requires separating two questions that are often treated as one: “does the model predict better?” and “does the company decide better?”. The first is answered with statistical error; the second with operating impact. When those questions are treated as equivalent, a project can report a technical win while the business outcome stays flat. A model may reduce WAPE while feeding exactly the same replenishment rule as before. If the decision process does not absorb the improvement, the accuracy remains trapped inside the model.
Where the chain breaks
A typical chain has four stages: data, forecast, decision and execution. Error can fall in stage two while the bottleneck remains in stage three. A team may create better weekly forecasts while procurement still places monthly orders using fixed rules. In that situation, much of the new signal arrives too late or is aggregated away before it can influence action.
The reverse can also happen. Forecasts may update frequently while operations cannot change. A supplier may require orders to be confirmed weeks ahead, staffing may be locked before the shift, or machine capacity may already be committed. Forecast quality also depends on whether the information arrives before the decision becomes hard to reverse. The first diagnostic therefore should not be another model metric. It should be a map from forecast to action: who receives it, when, under which constraints and which part of the decision is still adjustable.
Cost is asymmetric
Common metrics summarize error magnitude, but the business can pay very different costs depending on direction. Under-forecasting a critical item may create a lost sale or a production stop; over-forecasting may only add inventory. In another setting, excess is worse because the product expires or loses value quickly. A +10 error and a -10 error do not necessarily have the same economic meaning. That changes model evaluation. When operating costs are asymmetric, optimizing a symmetric metric can reward the wrong behavior. MAE and RMSE still provide useful statistical information, but they should sit next to a cost function that reflects the downstream decision.
The same principle applies across horizons. The error that matters is the error present when the decision is made. A nine-month forecast can be less accurate than a one-week forecast and still be valuable for strategic capacity. Applying the same tolerance to both can penalize the model solving the harder decision.
The aggregation problem
Averages create another source of false improvement. Total WAPE can fall because high-volume items improve while low-volume products with high margin or high criticality become worse. The aggregate describes the portfolio, but it can hide exactly the segments where error is most expensive. Hierarchy creates a similar issue. A forecast may be excellent at category level but weak by SKU, location or customer. If procurement acts by SKU and warehouse, strong national accuracy does not remove the operational problem.
The answer is not to create dozens of metrics without purpose. It is to segment by decision: criticality, margin, lead time, volatility or service target. Evaluation should use the same granularity as the action it is meant to improve. That makes it possible to identify where the model creates value and where another policy is required.
Design backwards from the decision
The more robust approach is to start with the decision. Define what must be decided, how frequently and under which constraints. Then determine which future information would help. Only after that should the team choose horizon, granularity, metric and model. This sequence prevents over-optimizing the predictor. The final point of accuracy is not always worth pursuing when replenishment policy, lead time or contractual capacity dominates the outcome. Sometimes the highest-value change is a better rule rather than a better algorithm.
A useful forecasting system should answer three questions: which statistical error does it reduce, which decision does it change and which operating cost does it improve? If it can only answer the first, it is still a prediction project. Accuracy is useful evidence; impact is the final test.