Your MAE Went Down and You Still Lost Money

Here are two order quantities for the same SKU, evaluated against the same demand distribution.

The first is better by mean absolute error. It is also better by mean squared error. It happens to be the mean of the demand distribution, and its median, and its mode. By every accuracy metric on the dashboard, 80 is the right answer.

It costs 33% more than ordering 100.

This is not a rounding artifact or a selectively chosen distribution. It is the normal case, and it follows from something we argued qualitatively in Beyond Accuracy: What Makes a Good Forecast?: a good forecast has to support a real decision. This post makes that concrete with a loss function.

The mean answers a question nobody asked

Train with squared error and you recover the conditional mean. Train with absolute error and you recover the conditional median. Both answer the same question: what is demand likely to be?

Nobody acts on demand. They act on an order quantity, a headcount, a provisioned instance count, a cash load. And the cost of being wrong is almost never symmetric.

Take a jacket that costs $\$30$ wholesale, sells for $\$70$, and clears at $\$20$ at the end of the season. Every unit of demand you failed to stock costs you the $\$40$ margin you didn't earn. Every unit you overstocked costs you the $\$10$ you lost clearing it.

$$C_u = 40 \qquad C_o = 10$$

Underage hurts four times as much as overage. Ordering to the mean treats those two as if they were the same thing, which is a decision about how often you are willing to run out. On this jacket the defensible stockout rate is one season in five. Ordering to the middle of the distribution accepts something much closer to one in two on a product where running out costs 4× a markdown. Nobody in the room chose that. It fell out of the metric.

That one in five rate isn't a guess. The fraction $C_u / (C_u + C_o)$, here $40/50$, or $0.8$, is called the critical ratio, written $\tau^{*}$. It is the answer to the classic newsvendor problem: stock the $\tau^{*}$ percentile of demand and you have priced the two mistakes against each other correctly. For this jacket that means ordering the 80th percentile, which is the same thing as accepting a stockout one season in five. Order the average instead and you have chosen a different number, without choosing it.

One caveat on scope, because it decides how you use everything below. The critical ratio is the answer to the newsvendor problem, which is a single ordering decision. In the original version a seller decides the night before how many newspapers to buy for the next day, the papers arrive in the morning and sell through the day, and whatever is left is discounted at the close because it is worthless tomorrow. The jacket fits the same shape: you commit to a quantity at the start of the season, sell through the season, and clear the remainder at $\$20$ near the end. One commitment, one demand realization, one salvage event, no chance to top up in between. Where you can reorder mid stream, the right model is a multi period inventory policy rather than a single quantile, and the critical ratio is only the first term of the answer.

Why safety stock formulas quietly fail

Most planners already know the mean isn't the order quantity. The standard patch is a safety stock buffer:

$\text{safety stock} = z \cdot \sigma \cdot \sqrt{L}$

where $z$ is the number of standard deviations matching the service level you're targeting, $\sigma$ the standard deviation of demand in one period, and $L$ the lead time counted in those same periods.

This is the right instinct with the wrong machinery. That formula assumes forecast errors are Gaussian, symmetric, and homoscedastic, with the same spread this week as next and the same shape for a fast mover as for a slow one. Real demand is skewed, intermittent, and heteroscedastic. Promotions fatten one tail and not the other.

The failure is worse than "slightly off," because the approximation degrades exactly where you're asking it to work. A critical ratio of 0.8 or 0.95 is a question about the upper tail, and the tail is where a Gaussian fitted to mostly central data is least trustworthy. You are extrapolating a symmetric bell into a region the data said was lopsided. The formula returns a confident number and the number is wrong in the direction that costs the most.

A quantile from a predictive distribution doesn't need any of those assumptions. You asked for the 80th percentile and you get the 80th percentile.

A worked example you can check by hand

Let demand take five values:

This distribution is deliberately symmetric. Its mean, median, and mode are all 80, so nothing here depends on skewed demand. With $C_u = 40$ and $C_o = 10$, we add one extra column: pinball loss, the standard score for a quantile forecast. It is a price list for mistakes: at quantile $\tau$ you pay $\tau$ for every unit you fall short and $1 - \tau$ for every unit you overshoot. At $\tau = 0.8$ that is 0.80 per unit short against 0.20 per unit over, so underestimating is penalized four times as hard as overestimating, and 4 is exactly $C_u / C_o$. Ordering 100 against demand of 60 costs $0.2 \times 40 = 8$; against demand of 120 it costs $0.8 \times 20 = 16$. Weight the five outcomes by their probabilities and you get the 6.0 in the table below.

Two things to notice.

The optimum is 100, the 0.8 quantile, exactly as the critical ratio predicted. Now multiply any pinball figure by $C_u + C_o = 50$: $6.0 \times 50 = 300$, and the same works on every row. That is not a coincidence. Pinball loss weights the two error directions by $\tau$ and the complement of $\tau$. At the critical ratio, those weights are exactly $C_u$ and $C_o$ divided by their sum. Undo the division and you are back to money, to the cent. Pinball loss at the critical ratio is not a proxy for business cost. Up to a constant it is business cost, and MAE is the special case $\tau = 0.5$, which assumes stockouts and markdowns hurt equally. No retailer believes that.

Then look at ordering 60 versus ordering 100. Identical MAE of 24. Costs of $\$900$ and $\$300$. MAE cannot see a 3× difference in money, because MAE was never measuring money. Pinball loss at 0.8 separates them 18.0 to 6.0, precisely 3:1.

As the critical ratio climbs, the penalty for ordering to the mean compounds:

At $\tau = 0.5$ the mean is exactly right. That is the only row where it is, and it describes a business where running out and marking down are equally painful.

Each curve is normalized to its own best achievable cost, so 1.0 is optimal regardless of the absolute dollars. Three curves, three different minima: 80, 100, 120. The dashed line marks the mean. Only the 0.50 curve bottoms out there. At a critical ratio of 0.95, ordering to the mean costs 4× the optimum.

Asking TimeGPT for the quantile you actually need

TimeGPT returns quantiles of the predictive distribution alongside the point forecast, and it gets them without assuming a shape. The uncertainty comes from conformal prediction: the model backtests itself on rolling windows of each series' own history, collects the errors it actually made, and calibrates the spread from those. Skew, intermittency, and a variance that moves week to week are priced in because they showed up in the residuals, not because someone chose a distribution that admits them.

You ask for them with the quantiles argument. Pass quantiles=[0.8] and you get back a column named TimeGPT-q-80, the demand level that 80% of the distribution falls below. That is the number the newsvendor asks for, and it arrives next to the point forecast rather than in place of it, so you can still report the mean while ordering to something else:

import pandas as pd
from nixtla import NixtlaClient

client = NixtlaClient()  # reads NIXTLA_API_KEY from the environment

cu, co = 40.0, 10.0
tau = round(cu / (cu + co), 2)  # 0.8, from the critical ratio

forecast = client.forecast(
    df=sales,  # columns: unique_id, ds, y
    h=1,  # one horizon step, matching the single ordering decision
    quantiles=[tau],
)

# This column is the order quantity.
forecast["order_qty"] = forecast[f"TimeGPT-q-{int(tau * 100)}"]

The newsvendor decision is made once, against the demand of the period that follows it, so the quantile you order to has to be the quantile of that one period, that's why you need to call the forecast with h=1. Aggregate the history to the frequency the decision runs at and ask for a single step.

Every item has its own critical ratio

That 0.8 belongs to the jacket. The umbrella sitting next to it in the same purchase order has a different margin and a different salvage value, so it has a different critical ratio, and the phone case a third. $C_u$ and $C_o$ are properties of the item's economics, not of the forecast, and they are already in your systems: unit cost, selling price, markdown or salvage recovery, and where they apply, the cost of expediting a replenishment or writing off an expired unit.

Almost every planning system flattens all of that into one service level applied across the whole catalog, usually 95%, set once and rarely revisited. That is the same mistake as ordering to the mean, moved up one level: a single number standing in for thousands of different tradeoffs. It overstocks the items that are cheap to run out of and starves the ones where a stockout costs several times a markdown, and no accuracy metric will ever surface it, because accuracy is not what went wrong.

Fixing it is a join, not a model. quantiles takes a list, so a single call returns every quantile the catalog needs, and each item reads the column its own ratio points to:

costs = pd.DataFrame({
    "unique_id": ["jacket", "umbrella", "phone_case"],
    "cu": [40.0, 6.0, 30.0],
    "co": [10.0, 4.0, 10.0],
})
# taus: 0.80, 0.60, 0.75
costs["tau"] = (costs["cu"] / (costs["cu"] + costs["co"])).round(2)

forecast = client.forecast(
    df=sales,
    h=1,
    quantiles=sorted(costs["tau"].unique()),
)

orders = forecast.merge(costs, on="unique_id")
orders["order_qty"] = [
    row[f"TimeGPT-q-{int(row['tau'] * 100)}"] for _, row in orders.iterrows()
]