New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
College hockey calibration (2026-10-21)
Scripts: calibration_options.py, calibration_forward_2026.py, plus their *_results.json.
Inputs: - Walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. model folds (2022, 2023, 2025; 2,799 non-tie games). - A point-in-time EloElo ratingA rating system, originally from chess, that moves a team up or down based on results and the strength of the opponent. replay (production rule: k=22.5, HFA 33.6). - Every candidate is fit only on earlier seasons.
What's wrong (raw, folds pooled)
Model - Under-predicts home teams: the 0.5–0.6 bin predicts 0.547 and observes 0.598. - Underconfident at the top: 0.6–0.7 predicts 0.639, observes 0.721; 0.7–0.8 predicts 0.745, observes 0.884. - Platt slope is ~1.2.
Elo - Overconfident: Platt slope 0.79–0.81. - Also under-predicts home teams: intercept +0.15 to +0.19 on the log-odds scale. HFA 33.6 Elo points is too low.
Held-out log loss, 2023 + 2025 scored
| log losslog lossA score for probability forecasts that punishes confident wrong answers harshly. Lower is better. | Brier | vs raw model (95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all.) | |
|---|---|---|---|
| Raw model (what the site serves) | 0.6445 | 0.2265 | — |
| Platt on model | 0.6414 | 0.2251 | −0.003 (−0.009, +0.002) |
| Raw Elo | 0.6299 | 0.2205 | −0.015 (−0.027, −0.001) |
| Platt on Elo | 0.6278 | 0.2196 | −0.017 (−0.029, −0.004) |
| Model + Elo, logistic | 0.6255 | 0.2185 | −0.019 (−0.029, −0.009) |
Plain Elo beats the GBM model out of sample.
Forward check (fit on all folds, applied to the DEPLOYED model's 2026-27 pre-game predictions)
Fit: p = σ(0.105 + 0.383·logit(p_model) + 0.631·logit(p_elo)).
| n | Deployed | Fix | Market | |
|---|---|---|---|---|
| All 2026 games | 88 | 0.647 | 0.620 (−0.027, CI −0.071..+0.019) | — |
| 2026 priced | 83 | 0.657 | 0.634 | 0.638 (fix − market −0.004, CI ±0.03) |
| 2025 priced (fit on 2022+23) | 150 | — | — | fix − market −0.013 (CI −0.040..+0.012) |
Verdict: - The recalibrated mix brings the model from 0.02–0.04 worse than the market to about level with it. - It is not an edge: every CI against the market spans 0. - The share of priced games where model and market differ by 5+ points falls from 64% to 51%.