MAE vs RMSE: Why Your Model Optimizes One Number and Your KPI Reports Another
MAE and RMSE both measure the size of forecast error in the units you forecast in, and RMSE is never the smaller of the two, because it squares each error before averaging so a handful of big misses dominate the score. The practical difference is aim, not sensitivity. A forecast that minimises MAE targets the median of the demand distribution; one that minimises RMSE targets the mean (Hyndman & Athanasopoulos, Forecasting: Principles and Practice, 3rd ed.). Use MAE when you want a typical miss you can explain in an operations meeting. Use RMSE when large misses cost more than proportionally, which is true wherever capacity, expediting or air freight is involved.
What is the difference between MAE and RMSE?
MAE is the average of the absolute errors, so a 100-unit miss counts exactly ten times a 10-unit miss. RMSE squares each error, averages the squares, then takes the square root, so that same 100-unit miss counts a hundred times as much before the root pulls the answer back into units. Both land in your forecast units, which makes each of them readable at item level and neither of them comparable across items of different size. Hyndman and Athanasopoulos put that plainly: measures based only on the error term "are therefore scale-dependent and cannot be used to make comparisons between series that involve different units."
The deeper split is what each one is optimal for. Hodson’s 2022 review in Geoscientific Model Development settles a very old argument by pointing out that neither metric wins in general: RMSE is optimal for normally distributed errors and MAE is optimal for Laplacian errors, so the right choice follows from what your error distribution actually looks like. Demand error in a promoted, seasonal, event-driven category is not Gaussian, and it is not Laplacian either. Most of us are picking the least wrong option, and it is worth saying so out loud rather than pretending the textbook answer transfers cleanly.
There is a longer-running argument underneath this. Willmott and Matsuura argued in 2005 that RMSE is an ambiguous measure of average error because it varies with the spread of the error magnitudes and with the square root of the number of errors, not just with average error size, and that dimensioned model comparisons should use MAE. Twenty years on, RMSE is still the default in most fitting routines. That tells you something about how much of metric choice is inherited rather than chosen.
What does the RMSE-to-MAE ratio tell you?
It tells you whether your error is spread evenly or concentrated in a few bad periods, and it is free, because you already calculate both numbers. RMSE can never be less than MAE, and it can never exceed MAE multiplied by the square root of the number of observations. That upper bound comes straight out of the Willmott and Matsuura argument above, and it matters practically: the ratio’s ceiling moves with your sample size, so a "high" ratio over six periods and a "high" ratio over 52 weeks are not the same number.
| Periods scored (n) | Ratio ceiling (√n) | Ratio near 1 means | Ratio near the ceiling means |
|---|---|---|---|
| 6 (a half-year of months) | 2.45 | Every period misses by roughly the same amount. A level or bias problem. | One period holds nearly all the error. Look for a promotion or an event. |
| 12 (a year of months) | 3.46 | Steady, diffuse error. Model structure or a missing driver. | One or two months dominate. Check the promo calendar before touching the model. |
| 52 (a year of weeks) | 7.21 | Consistent weekly noise, close to irreducible. | A handful of weeks carry the damage. Almost always events, rarely the algorithm. |
In our experience the useful reading is relative, not absolute. Run the ratio on the same item across two quarters, or across two items in the same category at the same n. A ratio that jumps between quarters is a change in the shape of the error, and shape changes are usually something that happened in the business rather than something that happened in the model.
We are deliberately not publishing a universal trigger number here. A threshold that travels intact from one business to another does not exist, and inventing one would make this page more quotable at the cost of being wrong in most of the places it got quoted. What does need to exist is your threshold. Fix your n, run the ratio across a representative slice of items for two or three quarters, and record where it sits when nothing unusual is happening. That spread is your baseline, and a reading that steps outside it has earned a look. Skip this and you end up in the meeting where everyone argues about whether 1.6 is high, which is unanswerable in the abstract and obvious the moment you have your own numbers in front of you. Having a threshold matters more than which threshold you land on.
Can MAE and RMSE pick different winners on the same data?
Yes, routinely, and one promotional spike is enough to do it. Below is a constructed example (not client data) built to make the disagreement visible: one item, six periods, one spike in period 5. Forecast A is a flat 105, a reasonable read of a normal week. Forecast B is a flat 140, hedged upward because the planner knows spikes happen and would rather carry cover.
| Period | Actual | Forecast A | Error A (F−A) | Forecast B | Error B (F−A) |
|---|---|---|---|---|---|
| 1 | 100 | 105 | +5 | 140 | +40 |
| 2 | 120 | 105 | −15 | 140 | +20 |
| 3 | 90 | 105 | +15 | 140 | +50 |
| 4 | 110 | 105 | −5 | 140 | +30 |
| 5 | 300 | 105 | −195 | 140 | −160 |
| 6 | 95 | 105 | +10 | 140 | +45 |
| Total | 815 | 630 | 840 |
Score both and the scorecard splits:
| Metric | Forecast A | Forecast B | Which wins |
|---|---|---|---|
| MAE | 40.8 units | 57.5 units | A |
| RMSE | 80.2 units | 74.2 units | B |
| WAPE (Σ|F−A| ÷ ΣActuals) | 30.1% | 42.3% | A |
| Bias (ΣA − ΣF) ÷ ΣF | +29.4% (under-forecast) | −3.0% (over-forecast) | B |
| RMSE ÷ MAE | 1.97 (ceiling 2.45) | 1.29 | diagnostic, not a score |
Positive bias here means we under-forecast, so Forecast A sold 29.4% more than it planned for. Same six periods, same actuals. MAE and WAPE say the flat 105 is clearly the better forecast. RMSE and bias say the hedged 140 is. Both readings are defensible. Which one you act on depends on whether the cost of that period-5 miss scales with its size (go with A) or scales worse than its size, because it means expedited freight, overtime, or a customer who stops calling (go with B).
Forecast A’s ratio of 1.97 against a ceiling of 2.45 is the tell. Most of its error lives in one period. No amount of model tuning fixes a promotion nobody told the planner about, and we have watched a lot of model tuning get commissioned to fix exactly that.
How do you read MAE and RMSE across a mixed portfolio?
Run them side by side and let the distance between them do your triage. The two metrics do not only disagree about winners, they point at two different species of problem. MAE describes the error that is present more or less everywhere, all the time. RMSE describes where a small number of gaps are being amplified. Read together they separate a systemic problem from a local one, which is the question you actually have when a portfolio-level accuracy number moves and nobody knows why.
This shows up most clearly in multi-category FMCG. Case sizes across such a portfolio have almost nothing in common, with ice cream and skin care moving in small cases while detergents and confectionery move in large ones. The demand patterns have nothing in common either. Some lines are heavily seasonal, some are close to pure impulse, some are steady monthly staples. Underneath all of that the supply chains can be entirely separate businesses, cold chain against pharma against non-edible grocery. One accuracy metric and one model choice stretched across that whole range is rarely a decision anybody made. It is usually whatever the implementation defaulted to on go-live and nobody revisited.
Reading both numbers together strips out the distortion that an overweight category puts on the portfolio view, and it tells you whether the damage is structural or sitting in one corner. Model selection can then follow the diagnosis. The real work behind this is cohorting: the closer you get to grouping product and customer cells that genuinely behave the same way, the better your model optimisation gets, because you are finally tuning against a population that shares a shape. What makes the two metrics worth the effort is that they get you most of the way there without commissioning an elaborate root cause analysis first. The gap between them is doing that job for you, for free, in numbers you already produce every month.
Which metric should a demand planner report?
Report the one that matches the decision the number drives, and accept that most businesses need two of them rather than one.
| Metric | What minimising it aims the forecast at | Comparable across SKUs? | Best used for | Where it misleads |
|---|---|---|---|---|
| MAE | The median of the demand distribution | No. It is in units and scales with volume | Item-level review; explaining a typical miss to a human | Pulls forecasts toward zero on intermittent demand |
| RMSE | The mean of the demand distribution | No. Same problem, and it moves with n as well | Model selection where big misses hurt more than proportionally | One bad period can condemn a decent model |
| WAPE | Volume-weighted total error | Yes | The portfolio KPI most businesses should be reporting | Hides a rotten long tail behind healthy A-items |
| Bias | Nothing. It measures direction, not size | Yes, as a percentage | Governance; spotting sandbagging and incentive problems | Says nothing at all about how big the error is |
| Scaled errors (MASE, RMSSE) | The same target as their parent metric | Yes, that is the entire point | Ranking forecast quality across items of very different size | Hard to explain in a demand review without a preamble |
Research picked a side on this years ago and industry mostly did not follow. The M5 accuracy competition, the largest public test of retail demand forecasting we have, scored entries on a weighted root mean squared scaled error across 30,490 Walmart product-store series rather than on any percentage metric, precisely because squared error copes with intermittent retail demand and scaling makes series of wildly different volume comparable (M5 accuracy competition: results, findings and conclusions, International Journal of Forecasting). Almost nobody reports anything resembling that in a monthly demand review. The distance between how forecasts are judged in research and how they are judged in your S&OP pack is worth sitting with for a minute.
Why is the MAE-optimal forecast for a slow mover zero?
Because the median of a mostly-zero demand series is zero, and MAE is minimised at the median. The generic explainers skip this completely. We are not the first practitioners to raise it, and it would be poor form to pretend otherwise: Nicolas Vandeput has made the same argument in a demand planning frame, pointing out that once more than half your periods carry no demand, "the optimal forecast is… 0!" He is right. What the rest of this section adds is what that finding does to a real portfolio once you stop filing it under curiosities. When you squash a demand distribution into one number, different error measures are minimised by different points on that distribution: absolute error by the median, squared error by the mean. It is a property of the measures rather than a rule of thumb, set out in Stephan Kolassa’s "Why the ‘best’ point forecast depends on the error or accuracy measure" (International Journal of Forecasting 36(1), 2020). Kolassa made the same point with retail count data in Evaluating predictive count data distributions in retail sales forecasting (IJF 32(3), 2016).
Take a C-item that sells nothing most weeks and two units occasionally. Median weekly demand is zero. So the forecast that minimises MAE is zero, every week, permanently. It will score beautifully on the accuracy dashboard. It will also tell replenishment to order nothing for an item that does, in fact, sell.
Teams run into this without ever recognising it. A machine learning model gets retrained on an absolute-error loss because someone reasonably decided outliers were distorting the old one, the forecast for the slow-moving tail collapses toward zero, the accuracy number improves, and the tail starts stocking out about six weeks later while everyone points at the dashboard and says the forecast got better. The model did precisely what it was asked to do. The asking was wrong.
This is not a shrinking corner of the problem either. We hit it again recently, and the honest read is that it grows from here. Portfolios keep moving toward hyper-customisation, which produces a tail that effectively never ends, which means the share of your catalogue carrying an intermittency problem gets bigger every year rather than smaller.
That is also the part of the range the forecasting vendors leave alone. The pitch is that a machine learning model does the heavy lifting on data preparation and hands back better accuracy, and across the fast-moving part of the portfolio that is often fair. The intermittent tail is a different animal and rarely gets tackled. More often it is written out of scope at the start, because a scope that excludes the hard population keeps the success criteria comfortably reachable. Read any accuracy improvement claim with that in mind, and ask which items were in the test set before you ask how big the improvement was.
When we dug into why our own tail behaved the way it did, two things came out, and neither is a forecasting problem in the way that phrase is usually meant.
The first is the ratio of minimum order quantity to demand. A good deal of intermittency is self-inflicted and gets designed in long before anyone forecasts anything, at the point where a product’s MOQ is set with no reference to how fast the thing actually sells. Nobody in engineering is thinking about a demand planner’s accuracy report in that meeting, and there is no reason they would be. It has nothing to do with how you measure, report or fix accuracy. It has a great deal to do with the numbers you end up staring at.
The second is more useful. What reads as irregular demand at the aggregate level is frequently a very orderly reorder-point pattern at the individual customer level. Several customers each reordering on their own predictable trigger, summed and viewed from above, look like noise. Catch the pattern customer by customer and a real chunk of the supposedly unforecastable population turns out to be forecastable after all.
Which points at a fix that is not a model. Bring the customer into the conversation through CPFR, or whatever collaborative planning arrangement you can actually get signed, and keep a named, maintained list of the items your data science team has declared out of scope. Those items still ship and still stock out, so somebody owns them either way. Far better that ownership is an explicit list with a method attached than a silent gap sitting behind a healthy-looking accuracy number.
For intermittent demand, select models on squared error or on a quantile loss aimed at the service level you actually promise customers, and keep MAE for explaining results to people rather than for choosing models.
Does your planning system optimize the metric you report?
Usually not, and in our experience almost nobody checks. The optimisation measure is a configuration field somewhere in the forecast profile, it gets set once during implementation by whoever was in the room, and it never gets revisited. SAP IBP, to take a system that documents its options publicly, lets you choose which error measure it uses when it compares candidate algorithms and settles on a best fit, and it publishes the set of measures available for that comparison, RMSE and MAPE among them (SAP IBP forecast error measures). The point is not that SAP gets it wrong. The point is that the choice exists, it carries real consequences, and it is usually made a long way from whoever ends up owning the KPI on the wall.
Meanwhile the number in the S&OP pack is WAPE or MAPE. So the engine picks the model that would win one contest and then gets graded on a different one.
The mismatch is not always material. It turns material when demand is spiky or intermittent, which is exactly the population where model choice matters most.
There is an older principle underneath this that deserves saying plainly. Closed-loop control, where the thing being measured feeds back into the thing doing the work, has been a cornerstone of robust machines and processes since Watt’s flyball governor regulated steam engines during the industrial revolution. It keeps reappearing in new costume. Recursive self improvement in large language models, which is where a good deal of frontier lab effort is currently pointed, is the same idea again.
Applied to forecasting it is simple enough. If the KPI a function is judged on is different from the measure the machine is optimising, the two paths may cross by accident, but they will diverge, and they will diverge hardest in the cases you care most about. So unify them wherever the system allows it, with the business looking at the same number the engine is graded against. Where the system will not allow that, the workable second option is a translatable KPI, one that explicitly adjusts for the divergence instead of pretending it is not there.
What has served us best is neither of those on its own. It is a model audit, at minimum once a year, run as a real month-long piece of work with data science genuinely in the room. The questions are narrow: what broke, how often did it break, and how much did each break cost. An improvement program comes out of the audit and touches only the areas that are dilutive to the KPI in question. Everything else gets left alone on purpose. That is a closed loop you built yourself, and it buys more forecast quality than another round of algorithm shopping.
Two things are worth doing before you next argue about model accuracy:
- Find out what the fit engine actually minimises. Open the forecast profile. If the documentation is vague, ask the vendor in writing and keep the reply.
- Re-rank last quarter’s models on the metric you report. If the ranking changes, model selection has been answering a different question from your KPI for as long as the system has been live.
So which one should you actually use?
Use RMSE, or a scaled version of it, to choose models, especially on intermittent or spiky demand. Report WAPE plus bias to the business, because one gives a volume-honest error size and the other gives its direction. Keep MAE in units at item level, where a planner needs an error they can hold up against a pallet quantity. Then write down which single one of those decides anything, because a scorecard carrying five metrics and no hierarchy is how forecasting arguments become permanent fixtures of the calendar.
We covered the percentage metrics and their failure modes in MAPE vs WMAPE vs WAPE, the direction half of the problem in the forecast bias formula and the sign trap, and the question of whether any of this beats a naive baseline in Forecast Value Add.
