New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
NCAAB Production-Model CLV Eval — 2026-07-13
First market-relative evaluation of the production NCAABPredictionEngine (college-models-review
Track 1; plan ~/.claude/plans/keen-bouncing-spark.md). Reuses backtests/stacking_eval/ncaab/
plumbing (closing-line pull, multiplicative devigdevigRemoving the bookmaker's cut from odds to recover the market's actual implied probability., name-match, scoring, ROIreturn on investmentProfit as a percentage of the money wagered. +2% means $2 profit per $100 bet., splits); new glue =
espn_id key bridge + date-capped per-fold retrain (honest walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time.) + literal CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure..
Replication gate — PASSED
Reproduced the stacking-eval incumbent's devigged-consensus Brier on the shared 906-game OOS set:
0.19770463768533647 == incumbent 0.19770463768533647
(the consensus Brier is bit-identical to experiment_output.json), proving the harness plumbing is
bug-free before the production-model arm is trusted.
Leakage controls
- Date-capped per-fold retrain (
DateCappedNCAABEngine.train_asof): each fold trains ONLY ongame_date < fold_test_week_start;_save_modelsno-op'd so fold models never overwrite production pickles. Point-in-time features unchanged (get_team_recent_stats(before_date),get_team_rating_asof(before_date)). - Key bridge = direct equi-join (espn_id == ncaab_games.game_id), all rows resolved.
Result — REJECT (market wins; honest negative, as the literature predicted)
| slice | n | model_acc | cons_acc | model_Brier | cons_Brier |
|---|---|---|---|---|---|
| overall | 935 | 0.6695 | 0.6770 | 0.2249 | 0.1987 |
| low-liquidity ≤3 books | 129 | 0.7132 | 0.6899 | 0.1944 | 0.1879 |
| high-liquidity ≥4 books | 806 | 0.6625 | 0.6749 | 0.2297 | 0.2005 |
| conference | 886 | 0.6727 | 0.6749 | 0.2244 | 0.2009 |
| non-conference | 49 | 0.6122 | 0.7143 | 0.2332 | 0.1593 |
- Brier gap model−consensus +0.0261, game-clustered 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. [0.0155, 0.0370] — excludes zero: the market beats the model on calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing..
- ROI at min_ev 1–5%: -6.2%, -6.9%, -8.3%, -8.9% — all negative.
- The model's prices diverge ~11pp from the close on its picks, but that divergence is overconfidence (worse Brier, negative ROI), not edge.
- Verdict: REJECT. Expected valid honest negative — LRMC only tied Vegas, the platform's own stacked model already lost on this exact dataset. Keep informational on /ncaab; not a signal.
Bonus finding — 5 Barttorvik features are provably inert (ADOPTABLE)
28-feature vs 33-feature walk-forward: byte-identical results (model_brier 0.224857 both) — the 5 Barttorvik diffs (CloudFront-blocked → constant) contribute exactly nothing. Safe to drop them + the wasted daily Barttorvik pull.
Production gap discovered (data fix recommended)
[redacted] has ZERO rows for season '2026' (2025-26 season) as of 2026-07-13 -- this harness ran against an isolated snapshot copy (backtests/ncaab_eval/ncaab_games_snapshot.db) with that season backfilled via ESPN's public scoreboard API. Production DB untouched. See REPORT.md for the promotion spec to close this gap for real.
Data: backtests/ncaab_eval/production_clv_results.json. Harness: production_clv.py + build_snapshot.py.