New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
Confidence Pick'em — Honest Rebuild (2018–2025)
Date: 2026-07-20 · Harness: backtest_pickem.py → backtest_pickem_results.json
· Sample: 2,119 completed regular-season games · Seed: 20260720
Why this was rebuilt
The app advertised, in text rendered to users:
"Optimized Hybrid ... Backtested 2018-2024: 73.8% of max points" (vs 72.5% always-favorite)
That number had no artifact. It appeared in four docstrings, nothing computed it, and the only script that ever produced it had been deleted from the tree (surviving in a stale worktree) and printed to stdout without writing results.
Two further problems surfaced on inspection:
- The claims contradicted each other.
app.py:1105said "72.5% points (beats 72.4% favorite-only)" — 72.5% is ours there.app.py:2566said "73.8% vs 72.5% always-favorite" — 72.5% is the baseline there. The same number was both the strategy and the benchmark. - The advertised strategy was never the backtested one. The recovered script contains only spread rules. There is no XGBoost hybrid in it.
Results — % of maximum confidence points, per season
| Strategy | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | ALL |
|---|---|---|---|---|---|---|---|---|---|
| always_favorite (baseline) | 72.3 | 71.2 | 74.7 | 70.8 | 72.1 | 70.2 | 75.7 | 72.1 | 72.39 |
| fade_tiny_favorites | 73.0 | 71.4 | 75.1 | 70.7 | 73.6 | 69.4 | 75.3 | 71.8 | 72.53 |
| exponential_confidence | 72.3 | 71.2 | 74.7 | 70.8 | 72.1 | 70.2 | 75.7 | 72.1 | 72.39 |
| tiered_confidence | 72.3 | 71.2 | 74.7 | 70.8 | 72.1 | 70.2 | 75.7 | 72.1 | 72.39 |
| coin_flip (floor) | 51.5 | 49.0 | 49.4 | 52.5 | 51.6 | 45.2 | 49.9 | 47.0 | 49.48 |
Straight-up accuracy is 66.4% for every favorite-based strategy; 48.9% for the coin-flip floor.
Nothing beats the baseline
Season-clustered 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. on the points-percentage difference:
| Strategy | Δ vs baseline | 95% CI | Verdict |
|---|---|---|---|
| fade_tiny_favorites | +0.15pp | [−0.31, +0.64] | no difference |
| exponential_confidence | +0.00pp | [+0.00, +0.00] | no difference |
| tiered_confidence | +0.00pp | [+0.00, +0.00] | no difference |
| coin_flip | −22.91pp | [−24.64, −20.98] | loses |
Two of the "strategies" were mathematically incapable of differing
exponential_confidence (spread^1.5) and tiered_confidence (bucketed
multipliers) score exactly +0.00pp — not approximately, exactly. Both are
monotonic transforms of spread magnitude, and confidence pick'em scores only
the ranking, not the values. A monotonic transform cannot change a ranking, so
these produce byte-identical picks and points to always-favorite in every week
of every season. They were never alternatives to the baseline; they were the
baseline wearing different arithmetic.
What the advertised number was
The best reproducible result is 72.53% (fade_tiny_favorites, not significant) against a 72.39% baseline. The advertised 73.8% is not achievable by anything tested — it sits ~1.4pp above every reproducible strategy.
Notably the older comment (app.py:1105: "72.4% favorite-only") matches the
measured baseline almost exactly. The 73.8% figure appears to be the inflated
one, and it was the one shown to users.
What is still untested
The model hybrid ("trust XGBoost when spread < 3") could not be evaluated at
all: it requires the model's archived week-by-week predictions and none were
ever kept — pickem_picks holds 0 rows. Reporting a hybrid figure without
them would repeat the exact failure this document exists to correct.
Actions taken
- The user-facing
strategy_descriptionno longer claims 73.8%. It now states the measured benchmark (72.4% of max points, 66.4% SU, 2018–2025). - All four docstring instances corrected.
- This harness and its JSON artifact are committed, so the number is reproducible rather than remembered.
The takeaway for pick'em
Always picking the Vegas favorite scores 72.4% of maximum points at 66.4% straight up, and eight seasons of spread-based variation produced nothing that beats it. The market's favorite is a hard baseline here for the same reason the devigged close is elsewhere on this platform. Any future pick'em claim needs to clear 72.4%, walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time., with an artifact.