ShelfSense

Retailers need to forecast how much of each item each store will sell, every day, for thousands of item×store combinations at once, get it wrong and you either run out of stock or waste money on inventory that doesn't move. ShelfSense is a portfolio project built on Kaggle's M5 Forecasting Accuracy competition (the Walmart demand-forecasting dataset), demonstrating production-grade forecasting engineering rather than a one-off notebook.

The dataset is genuinely hard: 30,490 series, and 68% of item-days sell exactly zero units. A model that isn't built for that will systematically over-forecast the long tail of slow-moving items.

Raw sales, calendar, and price CSVs are first validated against Pandera schemas (a library that checks data against a declared structure, types, ranges, nullability, and fails loudly if it doesn't match) before anything downstream touches them.

From validated data, a 38-feature matrix is built per store, lag features (past sales at fixed offsets), rolling statistics, calendar signals, price signals, and hierarchy features, and written out as per-store parquet batches.

Seven LightGBM model variants are trained on that feature matrix using Tweedie loss (a loss function suited to zero-inflated count data), each tuned with Optuna (automated hyperparameter search). The winning approach trains 28 separate models, one per forecast horizon day, rather than one model that recursively feeds its own predictions back in as input. The seven variants are then ensembled into the final submission.

The whole pipeline is orchestrated by Dagster (22 assets, 20 automated data-quality checks), experiments are tracked in MLflow, configuration is managed with Hydra, and the raw data itself is versioned with DVC, so any run can be reproduced exactly.

A validated feature pipeline feeds seven independently-trained model variants, each attacking the same 30,490 series from a different angle, ensembled into one submission.

Cross-cutting: Dagster orchestration (22 assets, 20 quality checks), MLflow experiment tracking, Hydra config, DVC data versioning.

1. Tweedie loss instead of RMSE, because 68% of the data is zero

   Standard RMSE (root mean squared error) treats every prediction error equally, but with 68% zero-inflated demand it pushes models toward over-forecasting the tail. Tweedie loss is built for exactly this shape of data, and switching to it measured a +0.02 WRMSSE gain, a real number, not a hunch.

2. 28 direct per-horizon models instead of one recursive model

   A recursive forecaster predicts day 1, then feeds that prediction back in as input to predict day 2, and so on, so early errors compound forward. Training a separate model per horizon day avoids that compounding entirely, at the cost of more models to train. The measured cost of going recursive instead was 11% worse WRMSSE.

3. One global model across all series, not one model per series

   Classical per-series models (like ETS, a smoothing-based method fit separately to each individual series) collapse on sparse categories with little history, the HOBBIES category scored 3.27 WRMSSE that way. A single LightGBM model trained across all series at once can borrow statistical strength from the busier series to help the sparse ones.

4. Schema validation at every persistence boundary, because it caught a real bug

   Pandera schemas are enforced everywhere data is written to disk, not just once at ingestion. This isn't theoretical rigor, it caught an actual NaN (missing-value) bug in production data before it silently corrupted downstream features.

0.5693, show source for Best private-leaderboard WRMSSE

Verified source

WRMSSE, best private leaderboard

• reports/leaderboard.md:233 ↗

commit 2cf85c6 · measured 2026-05-07

Best private-leaderboard WRMSSE · a 36% reduction against the naive-seasonal baseline below, that percentage is derived from the two, not printed in the leaderboard

0.8956, show source for Naive-seasonal (Seasonal Naive 28) reference baseline

Verified source

WRMSSE, Seasonal Naive 28 reference baseline

• reports/leaderboard.md:221 ↗

commit 2cf85c6 · measured 2026-05-07

Naive-seasonal (Seasonal Naive 28) reference baseline · the reference the model is measured against, same leaderboard, same commit

3.2663 → 0.6112, show source for HOBBIES category, per-series ETS → global LightGBM

Cited in provenance.md

`README.md:158-164`

verified against source 2026-10-10

HOBBIES category, per-series ETS → global LightGBM · roughly a 5x reduction on the sparsest, hardest category

competition winner: 0.520 · this repo's realistic ceiling without GPU compute: 0.53–0.55, show source for Honest ceiling

Cited in provenance.md

`README.md:234-256,389-393,15,405,111,375`

verified against source 2026-10-10

Honest ceiling · 111 unit tests, 60%+ coverage; SARIMA was abandoned after an out-of-memory crash at series 442 of 1,000

Four new model variants were added late in the project, and every one of them looked like an improvement, on the 28-day validation holdout used to pick a winner before submitting to Kaggle. Then the private leaderboard scores came back, and all four had reversed.

The clearest example: RMSE-MH scored 0.6699 on validation against the baseline's 0.6860, a clear win on paper. On the private leaderboard, RMSE-MH scored 0.6205 against the baseline's 0.5693, a clear loss. The direction of the result flipped depending on which window of time you measured it on.

The root cause wasn't a modeling mistake, it was a measurement mistake: the validation window and the private-leaderboard evaluation window aren't statistically exchangeable, they cover different periods with different demand patterns, so a model that fits one doesn't necessarily generalize to the other. The honest conclusion was that the harness itself was the bug, not the models it was rejecting or accepting. Walk-forward cross-validation (evaluating across multiple rolling time windows instead of one fixed split) became the project's top roadmap item as a direct result.

If you're choosing between models based on a validation split, this is the check worth running before you trust it: confirm the validation window and the real evaluation window are actually statistically exchangeable, because a harness that silently disagrees with itself is worse than no harness at all.