NHL Stacking v3 — De-Snooping Campaign Findings

Model: nhl_stacking_v3 (production's only currently-profitable model) Scope: honest, de-snooped performance measurement + hygiene-fix evaluation — NOT a search for new alpha Data: [redacted], 17,155 regular-season games, 2008–2024 (nhl_games join nhl_historical_odds) Harness: backtests/nhl_eval/nested_walkforward.py (verified sound by 3 independent adversarial passes, incl. 2 full byte-identical re-runs) No production files touched. No commits made. All new code confined to backtests/nhl_eval/.

Bottom line

The model does NOT show a statistically distinguishable edge under honest, correlation-corrected measurement.

Both CIs — for both the original headline and the de-snooped nested figure — overlap zero once same-game bookmaker-row correlationcorrelationHow closely two things move together, from -1 (opposite) through 0 (unrelated) to +1 (in lockstep). It does not by itself mean one causes the other. is properly handled. The nested point estimate being higher than the in-sample figure (+7.15% vs +4.28%) is not evidence of a surviving edge; it reflects which two of thirteen seasons happened to grade positive and carry the most (correlated) bookmaker rows, not a stable cross-season signal (see Profit Concentration and Winner's-Curse below).

Recommended headline replacement, everywhere +4.32% currently appears:

"Point estimate +4.32% in-sample / +7.15% nested-OOS (2012–2024), both statistically indistinguishable from zero once game/season correlation is respected — not a confirmed edge."

1. Replication check (sanity gate before proceeding to nested evaluation)

Fixed thresholds (min_edge=0.03, ev_cap=None/uncapped), matching production's train_stacking_v3.py:backtest_roi:

Important caveat (pre-existing, documented in the harness header, not introduced by this campaign): the +4.32% figure comes from an uncapped EV% threshold. Applying the live scanner's nominal 8% EV ceiling on top of the 3% floor does not reproduce +4.32% — it gives +4.56% (2012–2024), because the cap prunes a net-negative high-EV tail. This harness replicates faithfully with ev_cap=None since that is what +4.32% was actually computed from.

In-sample per-season breakdown (2010–2024, min_edge=0.03, ev_cap=None)

Note the huge bet-count jump in 2022–2024 vs earlier seasons: this is bookmaker-row density (up to 22,805 book-rows over 1,190 games = 19.2x/game in 2022), not more games — this is the mechanical driver behind the naive-CI invalidity and the profit-concentration finding below.

2. Nested walk-forward OOS (the honest, de-snooped protocol)

Design: for each test season 2012–2024, select (min_edge, ev_cap) by grid search over a game-deduplicated validation pool (one row per (game, side), VAL_GAME_FLOOR=100 distinct games — this replaced the earlier VAL_BET_FLOOR=100 book-rows floor that let bookmaker-row duplication silently dominate validation-window selection), then settle bets in the actual held-out test season at the selected thresholds.

Aggregate

Per-season selected thresholds and outcomes

6 unique (min_edge, ev_cap) pairs selected across 13 seasons (previously, pre-fix, 2022/2023/2024 all converged on the identical tightest cell (0.06, 0.10) — that convergence is now GONE after deduping the validation pool to distinct games, confirming it was a mechanical artifact of book-row duplication, not a real regime signal).

Winner's-curse check: mean gap between selected-cell validation ROI and realized test-season ROI = +7.13pp (median +9.45pp), val>test in 8 of 13 seasons. This is a direct, quantified sign that the grid-search selector chases validation-window noise rather than identifying a stable, forward-predictive threshold regime — even after removing the book-duplication artifact from the selection mechanics.

Profit concentration (the core honesty finding)

• 2022 + 2023 alone supply $22,521.72 of $22,616.76 total nested profit = 99.6%.
• The remaining 11 seasons combined: 963 bets, profit $95.03, ROI = +0.10% — statistically indistinguishable from flat.
• Those same two seasons carry the heaviest, most correlated bookmaker duplication: 2022 = 22,805 book-rows / 1,190 games = 19.2x/game; 2023 = 16,897 / 1,229 = 13.7x/game — versus ~1x/game in the earlier era.

Conclusion: the nested aggregate point estimate (+7.15%) is largely a reflection of which two book-inflated seasons happened to grade positive, not a stable cross-season edge.

3. Why the naive CI is invalid, and what the valid ones say

The 2022–2024 era has 8–24 bookmaker quotes per game; each is currently treated by the production backtest definition as a separate "bet" (this is a pre-existing, documented property of train_stacking_v3.py:backtest_roi, independently reproduced by backtests/kelly_sim/build_nhl.py, whose own FINDINGS.md already flags "NHL rows per-book (correlated)" as a caveat — this campaign did not change that bet definition, only the statistical inference layer around it, to preserve comparability with the number being de-snooped).

Treating those duplicated same-game rows as independent trials (the naive bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval.) massively understates true uncertainty:

Both valid methods roughly double-to-quadruple the naive CI width and, critically, both cross zero for both the in-sample and nested figures.

4. Hygiene-fix experiments (all rejected — do not adopt)

4a. Rest-day feature hygiene (rest_hygiene_experiment.py)

239 games (concentrated in 2009/2013/2019/2021–2024 lockout/COVID/schedule-compression years) have implausible rest gaps up to 311 days, unclipped in production's add_rest().

Under the honest nested protocol, ROI monotonically worsens as hygiene is added, and variant (c)'s risk of a negative outcome roughly quadruples (6.8% → 31.1%). Brier-vs-devigged-consensus is flat to 4 decimals across all variants and both anomaly seasons (2013, 2021) — clipping has no detectable calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing. benefit; it only changes which bets get selected downstream via the threshold mechanism (fewer bets: 3,163 → 2,503 → 2,883).

Pre-registered house bar (adopt only if b/c matches-or-beats a AND doesn't degrade anomaly seasons): FAILS on both counts for both variants. Verdict: reject.

4b. Devig convention consistency (devig_consistency.py)

Compares production's multiplicative devigdevigRemoving the bookmaker's cut from odds to recover the market's actual implied probability. against Shin, under the model's real gating mechanism (edge floor is devig-dependent, EV ceiling is not).

Model and market Brier are identical to 4 decimals under both conventions — all ROI difference is bet-selection mechanics, not signal quality. The naive full-sample "pass" for Shin is entirely manufactured by 2022+2023 concentration (84.0% of multiplicative's nested profit and 132.2% of Shin's — i.e. Shin's other 11 seasons are net negative). Also confirmed by code inspection: Shin is currently dormant in the live nhl_stacking_scanner.py alert path (calls evaluate_h2h() without a pinnacle_fair_map) — there is no live inconsistency to fix today.

Verdict: reject. No change to devig convention.

4c. Snapshot-timing audit (snapshot_timing_audit.py) — diagnostic, not adopt/reject

Confirms, by source-code provenance (SBR archive's own labeled "close" columns for 2008–2021; The Odds API historical-snapshot endpoint querying "as of commence_time" for 2022+) and independent cross-validation against [redacted]'s genuine live-poller history (174 games, 672 book-pairs, Feb–Mar 2026): nhl_historical_odds is a closing-price dataset, both eras. 64% of matched pairs are individually closer to the closing proxy vs 21% closer to an early-window proxy; mean prob-space distance to close is 2.3–2.6x smaller than to early.

Honest gap this exposes: production's live scanner fires alerts up to 4 days pre-game. Early-window prices differ from closing by a median 1.46pp (up to >10pp) in fair-probability space — large relative to the 3–6% min_edge gate. The backtested ROI figures (+4.32% / +7.15%) are a closing-line entitlement, not an estimate of what the live 4-day-to-2h alert cadence actually realizes. This is currently unmeasured (production doesn't store alert-time prices joinably). Recommended as a new, low-risk logging addition (see production changes).

4d. Deployed-artifact drift (deployed_artifact_drift.py) — diagnostic

Deployed pickle provenance confirmed clean: exactly reproducible (coefficients identical to floating-point exactness) from current DB via the documented training script. Zero prediction divergence found — but only because nhl_team_game_xg has no season-2025 rows at all, so "train on all data" is silently capped at season 2024 by a data-collection gap, not by design. A sanity check (refitting on season<2024, which does drop 1,313 games) confirms the harness correctly detects drift when the row set actually changes (max_abs_diff=0.0092), so the zero-divergence result above is a real finding about the pipeline, not a broken comparison.

Verdict: reject (no drift, no alpha) but flags a real ops item: backfill nhl_team_game_xg for season 2025 before the next retrain.

5. What the evidence does NOT support

• Does not support citing +4.32% (or +7.15%) as a confirmed, statistically real edge — both CIs overlap zero once correlation is respected.
• Does not support any change to live min_edge/ev_cap thresholds — the nested selector shows winner's-curse behavior (+7.13pp mean gap, val>test in 8/13 seasons), i.e. no forecasting skill to act on.
• Does not support adopting rest-day clipping — nested ROI strictly worsens, and no calibration benefit shows up in Brier.
• Does not support switching to Shin devig — clean-season performance is strictly worse (-7.3pp) and the naive full-sample pass is a concentration artifact; also currently dormant/non-issue live.
• Does not support treating 2022–2023 performance as representative of anything going forward.
• Does not support a definitive "the edge is fake, shut it down" conclusion either — P(ROI<0) is 6.8–16.7%, genuinely inconclusive, not a disproof.

6. Recommended actions

1. Documentation-only: replace "+4.32%" everywhere with the honest de-snooped statement (see Bottom Line). Zero production risk.
2. No threshold changes. Keep min_edge=0.03 / current ev_cap as-is; do not act on nested grid-search selection (winner's-curse, no skill demonstrated).
3. No feature changes. Reject rest-day clipping.
4. No devig changes. Reject Shin switch; note it's dormant for this model anyway.
5. New, low-risk instrumentation: log alert-time price alongside realized closing price going forward, to eventually measure true live-realized CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure./edge vs. the closing-line backtest entitlement.
6. Ops hygiene, not urgent: backfill nhl_team_game_xg for season 2025 before the next scheduled retrain, so "train on all data" stops silently excluding the newest season.

No production files were modified in the course of this campaign; no commits were made. Full machine-readable evidence: backtests/nhl_eval/nested_results.json, rest_hygiene_results.json, devig_consistency_results.json, snapshot_timing_results.json, deployed_artifact_drift_results.json.