Demand Forecasting: When the Simple Model Beats the Sophisticated One

The four original scenarios (delay, review, freight, RFM) never touched real time series, they only used month as one feature among others. This chapter closes that gap: predicting how many orders arrive per month. Full executed notebook, public on Colab.

The series, and why I trim the edges

Chapter 1 already showed orders per month, but with nearly empty edges, September/October/December 2016 (4, 324 and 1 order) and September/October 2018 (16 and 4). That's not a demand drop, it's the dataset starting and ending mid-month. For real forecasting I only use January 2017 through August 2018, 20 full months.

serie_completa = orders.groupby(orders['order_purchase_timestamp'].dt.to_period('M')).size()
serie = serie_completa['2017-01':'2018-08']

df = serie.reset_index()
df.columns = ['mes', 'pedidos']
df['t'] = np.arange(len(df))
df['mes_num'] = df['mes'].dt.month
df['sin'] = np.sin(2 * np.pi * df['mes_num'] / 12)
df['cos'] = np.cos(2 * np.pi * df['mes_num'] / 12)

groupby(...dt.to_period('M')).size() counts orders per calendar month across the whole series, and the slice ['2017-01':'2018-08'] keeps only the 20 full months. t is a simple counter (0, 1, 2...) that becomes the linear regression's trend axis further down. sin/cos of the month is how I give the model cyclical seasonality without creating a per-month dummy (which would need 11 extra parameters for only 20 training points, a recipe for overfitting): January and December end up close together in this circular representation, the way neighboring months should, something a regular dummy wouldn't capture.

Three candidates, tested on the last 4 real months

Instead of blindly extrapolating, I set aside May through August 2018 as a test set and compared three approaches: naive (repeats the last known value), seasonal naive (repeats the same month from the previous year), and a linear regression with trend plus cyclical seasonality (sine and cosine of the month, not a per-month dummy, which with only 20 points would blow up in parameters for too little data).

n_holdout = 4
treino = df.iloc[:-n_holdout]
teste = df.iloc[-n_holdout:]

pred_ingenuo = np.array([treino['pedidos'].iloc[-1]] * n_holdout)

pred_sazonal = []
for _, row in teste.iterrows():
    mes_ano_anterior = row['mes'] - 12
    pred_sazonal.append(serie_completa.get(mes_ano_anterior, treino['pedidos'].mean()))
pred_sazonal = np.array(pred_sazonal)

modelo_linear = LinearRegression()
modelo_linear.fit(treino[['t', 'sin', 'cos']], treino['pedidos'])
pred_linear = modelo_linear.predict(teste[['t', 'sin', 'cos']])

y_real = teste['pedidos'].values
comparacao = []
for nome, pred in [
    ('Ingênuo (último mês)', pred_ingenuo),
    ('Sazonal ingênuo (mesmo mês, ano anterior)', pred_sazonal),
    ('Linear + sazonalidade cíclica', pred_linear),
]:
    mae = mean_absolute_error(y_real, pred)
    mape = mean_absolute_percentage_error(y_real, pred)
    comparacao.append({'model': nome, 'mae': round(float(mae), 1), 'mape': round(float(mape), 4)})

iloc[:-n_holdout]/iloc[-n_holdout:] split off the last 4 months (May through August 2018) as test, the rest becomes training. pred_ingenuo is literally a constant, the last training value repeated 4 times. pred_sazonal looks up, for each test month, the same month one year earlier (row['mes'] - 12 on a monthly period already does that subtraction correctly), falling back to the training mean if that month isn't in the full series. modelo_linear.fit(treino[['t', 'sin', 'cos']], treino['pedidos']) trains only on the 16 training rows, and predict on the 4 test rows, the same fit/predict pair that already showed up in every regression scenario back in Chapter 4, just with time features instead of the earlier tabular ones. mean_absolute_error and mean_absolute_percentage_error compare each forecast against the real value across the 4 test months.

The simplest model won, and that's this chapter's finding

The naive model is off by 478 orders on average. The linear model with trend and seasonality, the "serious" candidate, is off by 2,166.5, almost 5 times worse. The explanation is in the series' shape: 2017 had strong growth (November is Black Friday), and the linear model learned that slope and extrapolated continued growth into May-August 2018. Except the base had already leveled off well before that (6,873 in May, 6,167 in June, 6,292 in July, 6,512 in August, an almost flat line). The "sophisticated" model predicts growth where there's no more growth. The naive one, which assumes nothing about trend, simply doesn't make that bet.

Loading real data...

The final forecast, and why it's a flat line

Since the naive model genuinely won on the test, that's the one I use to extrapolate the 3 months following the dataset's end, September through November 2018. The forecast is a flat line at 6,512 orders (the last known real value), with an uncertainty band that grows with the square root of the horizon (the further out, the more uncertain), computed from the model's own real error on the test set.

residuos_holdout = y_real - pred_ingenuo
std_holdout = residuos_holdout.std()

ultimo_valor = int(df['pedidos'].iloc[-1])
futuro_meses = pd.period_range(df['mes'].iloc[-1] + 1, periods=3, freq='M')

previsao_futura = []
for i, mes in enumerate(futuro_meses, start=1):
    largura = std_holdout * np.sqrt(i) * 1.96
    previsao_futura.append({
        'month': str(mes),
        'predicted': ultimo_valor,
        'lower': round(max(0, ultimo_valor - largura), 1),
        'upper': round(ultimo_valor + largura, 1),
    })

residuos_holdout is the naive model's real error on the 4 test months (predicted minus real), and std_holdout is those errors' standard deviation, the ruler for "how wrong" the model usually is. The for loop walks the 3 future months (i from 1 to 3), and largura = std_holdout * np.sqrt(i) * 1.96 is the uncertainty band: multiplying by the square root of i makes the band grow slower than linearly (the error 3 months out isn't 3 times the 1-month error, it's √3 times), and 1.96 is the standard multiplier for a roughly 95% interval assuming normally distributed error. max(0, ...) on the lower bound just keeps the band from turning into a negative order count, which wouldn't make sense.

A flat line as an "AI" forecast sounds anticlimactic, but it's the honest answer 20 months of data give. Predicting continued growth with no evidence it continues is the kind of mistake that looks good on a trend chart and bad in practice.

Wrapping up the chapter (and the project)

This chapter's lesson generalizes to the other five: a more sophisticated model isn't better by default, it's only better if its assumptions match what the data is actually doing. The linear model assumed continued growth, the base stopped growing, the linear model lost. It's the same lesson as Chapter 4 (good AUC isn't a good threshold) and Chapter 5 (statistically optimal K isn't rich segmentation), told a new way: a good training-fit metric doesn't guarantee a good decision going forward.

With that, the Olist project's six technical chapters are closed. If you haven't seen it yet, Chapter 0 is the interactive showcase of all of this, no prose at all, just for playing around.