College Eval Tracks 1+2 (NCAAB + CFB) — Synthesis, 2026-07-13

Verification method: independently re-opened the artifact files each build points to (walkforward_regen_results.json, replicate_spike_gate_result.json for CFB; production_clv_results.json for NCAAB) and diffed the build's prose numbers against the JSON on disk. CFB matched exactly everywhere spot-checked. NCAAB's checker-reported mismatch was reproduced independently from the artifact — confirmed, not just accepted on the checker's word.

Track verdicts

CFB-2b (walk-forward /cfb/performance regen) — TRUSTWORTHY

Checker verdict: sound, replication confirmed, only two minor (non-blocking) issues. Independently re-verified: replicate_spike_gate_result.json gate 1 is an exact match to the incumbent spike backtest (n=3880, acc=0.697423, brier=0.197621, roi=-5.3088%, n_bets=2499 — pass: true), and every pooled/per-season number quoted in the build report (n=4332 pooled acc=0.7089/brier=0.1892/logloss=0.5659; ROIreturn on investmentProfit as a percentage of the money wagered. +2% means $2 profit per $100 bet. baseline -5.37% CI[-10.32%,-0.29%]; ROI platt -10.44% CI[-16.26%,-4.59%]; market-side replication acc 0.7352 vs 2a's 0.7342) is bit-for-bit present in walkforward_regen_results.json. Leak verification file shows all_folds_pass: true.

Honest story: The production CFB model, retrained walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. with correct pregame-only history, ties the market on nothing and loses to it everywhere — model trails devigged closing ML by 5.8-5.9pp accuracy and ~0.033 Brier, loses even wider on spread, and shows a statistically significant negative flat-stake ROI (CI excludes zero) against devigged closing lines. This is the expected, literature-consistent honest negative, and it corrects a genuine methodological bug (the live retrodiction's range(season-window, season+1) includes the season it's scoring, i.e. train-on-test-season leakage) without which the existing /cfb/performance numbers cannot be trusted. Interestingly the leak fix did not flatter the old number — season-2024 walk-forward (acc 0.7125) actually beats the leaked retrodiction's own season-2024 number (acc 0.6929), so the case for replacing it rests on methodology, not on the old number having been inflated.

Must not ship as-is: nothing blocking. Two minor flags only: (a) app/sports/cfb/models/trained/ directory mtime changed during the session while all three pickle files' content-mtimes and git status are unchanged — almost certainly benign (no production model was overwritten) but worth a 30-second owner glance; (b) the "second replication gate" (market-math cross-harness check) is informal/self-consistency, not a strict pre-registered incumbent reproduction like gate 1 — already labeled that way in the build's own docstring, so this is a documentation nuance, not a defect. Also note the build's own disclosed operational caveat: a second concurrent process was hammering the same script/db during the session; the reported numbers come from one verified-isolated foreground run (leak-checks ALL PASS, in-memory count == persisted count == 4332) snapshotted before the race — treat walkforward_regen_results.json as authoritative and re-run in isolation if the live cfb_walkforward.db looks inconsistent when inspected.

Promotion steps (checker confirmed sound + replication confirmed → cleared to hand to the owner as an ordered spec; full detail in backtests/cfb_eval/PROMOTION_SPEC_2b.md, none of it applied): 1. Copy walkforward_regen.py's logic into scripts/regen_cfb_walkforward.py, retarget OUT_DB to production [redacted], writing to a new cfb_predictions_walkforward table that coexists with (never overwrites) the legacy cfb_predictions retrodiction rows. Run by hand once to backfill 2021-2025. 2. Attach that regen to whatever cadence already drives CFBPredictionEngine's weekly retrain (grep the existing scheduler entry point — this is not a nightly job; only re-run when a season completes or the model changes). 3. Apply the _compute_cfb_perf_data() diff in the spec doc: adds a new walkforward key (from cfb_predictions_walkforward, keyed off home_win_prob_platt/moneyline_correct_platt/spread_correct/total_correct) without touching or removing the existing overall/by_season/settled_predictions/dataset keys. 4. In templates/cfb/performance.html, promote the walk-forward block to the primary/default view; demote the existing retrodiction block to a collapsed "legacy retrodiction, not walk-forward" section rather than deleting it. 5. Deploy: kill -HUP $(cat logs/[redacted].pid); smoke-test /cfb/performance and /cfb/api/performance/full for 200s. 6. Explicitly deferred (flagged, not spec'd, different risk profile — live-serving-path change): CFBPredictionEngine.predict_game's own history_seasons = range(season-window, season+1) off-by-one could also be fixed in place for the live upcoming-predictions page. Not part of this promotion; owner's call, separate task.

NCAAB (production engine vs devigged consensus CLV) — NOT TRUSTWORTHY YET (as reported)

Checker verdict: flawed. I independently reproduced the checker's critical finding by opening production_clv_results.json directly: the build's headline "numbers" text (n=906, model_acc 0.6600, cons_acc 0.6788, model_ll 0.6400, cons_ll 0.5745, model_brier 0.2229, cons_brier 0.1977, min_ev-1% ROI -8.24% n=834) does not match the artifact. The artifact says: n=935, model_acc=0.66952, cons_acc=0.67701, model_ll=0.64540, cons_ll=0.57715, model_brier=0.22486, cons_brier=0.19873, min_ev-1% ROI -6.18% n=871 — confirmed exactly matching the checker's own independent rerun, so this is not a disagreement between two reviewers, it's a documented fact of the file on disk.

Two further checker findings I did not re-derive from scratch but which are internally consistent with the artifact and not contradicted by anything I read: (a) the 935-vs-906 population isn't a rounding difference, it traces to a real ET-vs-UTC date-attribution bug (854/1568 matched games' dates differ by exactly -1 day between the two harnesses' date sources); (b) the "5-fold walk-forward" retrain is a no-op for this particular season/window — every fold trains on an identical 8,468-row set (seasons=['2024','2025'], train_window=2 never excludes anything for an as-of-date inside Feb-Mar 2026) — real for the incumbent's own walk_forward but not for this new production-engine harness in this window, which the report presents as if genuine per-fold retraining occurred.

Honest story: On the numbers that are actually in the artifact (not the ones quoted for review), the production NCAAB engine still loses to devigged consensus on every headline metric — accuracy 0.6695 vs 0.6770, log losslog lossA score for probability forecasts that punishes confident wrong answers harshly. Lower is better. 0.6454 vs 0.5771, Brier 0.2249 vs 0.1987 — so the substantive REJECT conclusion (production model ties-or-loses to the market, the expected honest-negative outcome) is very likely directionally correct and independently reproducible either way. But the specific figures an owner would read off this build's prose are wrong, the OOS population silently differs from the incumbent's own 906-game population it claims to match, and the "5-fold" framing overstates what was actually tested for this window. None of that changes the direction of the finding, but it means the report as currently worded cannot be trusted as a numeric record and should not be cited or promoted until corrected.

Must not ship: the build's prose "numbers" block, as written, must not be copied into /cfb-style dashboards, memory notes, or any downstream doc — it is not what the code produces. Use backtests/ncaab_eval/production_clv_results.json directly (or a corrected REPORT.md regenerated from it) as the source of truth instead. Also do not present the 5-fold table as evidence of genuine walk-forward retraining for the Feb-Mar 2026 window without the caveat that all 5 folds are model-identical here.

Promotion steps: NONE. Per instructions, promotion steps are only issued where the checker confirmed the build sound + replication confirmed; NCAAB's checker verdict is "flawed." The replication gate itself (reproducing the incumbent's cons_brier=0.19770463768533647 on n=906) is solid and independently reconfirmed — that specific sub-claim can be trusted — but the production-model arm's report as a whole cannot yet be promoted or cited.

Before this track can be promoted, the owner should have the harness re-run and its REPORT.md/prose regenerated from production_clv_results.json (not hand-transcribed), with the date-attribution bug fixed (pick one canonical date source, ideally the incumbent's, and rerun to actually reproduce n=906), and either fix the fold-boundary logic so folds genuinely differ for this window or relabel the section honestly as a single static cut rather than "5-fold walk-forward."

One piece that IS independently trustworthy and separable from the rest of the NCAAB build: the Barttorvik 33-vs-28-feature inertness check. I opened production_clv_results.json's feature_check_28_vs_33 block directly: predictions with and without the 5 Barttorvik-derived features are bit-identical (predictions_bit_identical: true) on the full n=935 evaluation set — stronger and more direct evidence than the build's own text describes (which cited a season-2025 n=526 spot-check instead of pointing at this bit-identical result). This is also consistent with the pre-existing memory note (data-sources-roadmap.md, 2026-07-05: "Barttorvik inert (CloudFront 403 from this server)"), i.e. a third independent confirmation. Because zero-variance features provably cannot receive a nonzero XGBoost split, this specific sub-finding does not depend on the rest of the harness's disputed numbers and can be promoted on its own:

1. Remove the 5 BARTTORVIK_FEATURE_NAMES columns (adjoe_diff, adjde_diff, barthag_diff, tempo_sum, barttorvik_missing) from FEATURE_NAMES in [redacted] (33 base features would remain).
2. Retrain the moneyline[redacted] XGBoost models on the reduced feature set via the existing scripts/retrain_ncaab_models.py.
3. Disable/remove the daily Barttorvik pull cron (app.sports.ncaab.data.barttorvik.pull_current_ratings — locate via [redacted] grep, not identified in this session).
4. Defense-in-depth: re-verify variance is still zero on fresh season-2026 data once the DB backfill (below) is applied, before actually dropping — cheap re-check, not because failure is expected.

Separate, higher-priority blocker for NCAAB regardless of the above: [redacted]::ncaab_games had zero rows for season 2026 as of the build's run; a live re-check during this synthesis shows season 2026 now has 6,318 rows (range 2025-11-03 to 2026-04-06), i.e. something backfilled it after the build ran (build's own artifact was timestamped 01:41, DB mtime 02:18 — later). The owner should re-verify current DB state before assuming the backfill promotion step is still needed — it may already be done, possibly via a different path than the one the build specified (import_date_range). Either way, the production-engine arm's numbers in this build were run against a synthetic, backtests-local reconstruction of history keyed by normalized display name (not the DB's real ESPN-abbreviation keying), so even a fixed DB does not make this specific harness run "faithful to the live DB" retroactively — a fresh re-run against the real, now-populated DB is needed before trusting any production-arm number for this window.

Full per-arm tables (from the verified artifacts)

CFB — pooled 2021-2025, walk-forward (n=4,332)

CFB per-season (model baseline / market_ml, accuracy): 2021 n=849 0.7267 (no ML-line coverage); 2022 n=854 0.6663, mkt(n=676)=0.7101; 2023 n=868 0.7131, mkt(n=711)=0.7511; 2024 n=873 0.7125, mkt(n=725)=0.7421; 2025 n=888 0.7252, mkt(n=775)=0.7690.

CFB ROI vs devigged market ML, flat @2% min-edge, game-clustered 95% bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval. CI (n=10k draws, seed 42):

Replication gate 1 (vs experiments/cfb_spike.json): n=3880, acc=0.697423, brier=0.197621, ROI=-5.3088%, n_bets=2499 — exact match, pass: true.

Replication gate 2 (market-math cross-harness, 2022-2024 overlap, informal): market_ml acc 0.7352 (n=2088) vs 2a's 0.7342 (n=2107), brier 0.17598 vs 0.17631; market_spread acc 0.7552 (n=2516) vs 2a's 0.7548 (n=2541), brier 0.16031 vs 0.16032 — consistent within ~0.1pp / ~0.0003 Brier given the deliberate population-size difference (TRAIN_WINDOW 5 vs 4).

NCAAB — production engine vs devigged consensus, walk-forward (n=935, per the artifact — NOT the 906 quoted in the build's prose)

Diff CIs (model − consensus, game-clustered bootstrap): accuracy diff -0.0075 [-0.0353, +0.0203] (not significant); logloss diff +0.0683 [+0.0431, +0.0941] (significant, model worse); Brier diff +0.0261 [+0.0155, +0.0370] (significant, model worse).

Liquidity split:

Conference split:

ROI (flat stake, best closing price, backtest-only), game-clustered bootstrap 95% CI:

Literal CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure. (model's own-side prob minus devigged consensus own-side prob), game-clustered bootstrap: point +10.89pp, 95% CI [+10.04pp, +11.77pp] — positive but explicitly not evidence of value; it co-occurs with a robustly worse Brier score, i.e. reflects model overconfidence, not edge.

Barttorvik 33-vs-28-feature check: predictions bit-identical with the 5 features removed (predictions_bit_identical: true, n=935) — independently confirms the features are inert, consistent with prior memory (data-sources-roadmap.md 2026-07-05).

Replication gate (incumbent devigged-consensus Brier on n=906): reproduced exactly — cons_brier=0.19770463768533647 — independently reconfirmed by both the build and this synthesis review.

Bottom line for the owner

• CFB-2b: ship the promotion spec. Sound, replicated, honest negative (market wins), fixes a real leakage bug in the live retrodiction. Only trivial follow-ups (directory-mtime glance).
• NCAAB: do not cite this build's prose numbers or promote its harness output yet. The REJECT direction is almost certainly right (production loses to devigged consensus, matching house priors), but the specific figures reported for review are demonstrably wrong versus the artifact, the OOS population silently drifted from the incumbent's own 906 games via a timezone bug, and the "walk-forward" framing overstates what happened in this window. Re-run with the date-source bug fixed and the report auto-generated from the JSON before trusting or promoting any number from this build. The one exception: the Barttorvik-inert finding is independently solid (bit-identical predictions) and safe to promote on its own, once paired with the separate DB-backfill housekeeping already flagged (and possibly already resolved — verify current DB state first).

Addendum — clean NCAAB re-run 2026-07-13 (resolves the "not trustworthy yet" flag)

The NCAAB harness was re-run in isolation (workflow finished, no competing sandbox process) against the now-populated production [redacted] (season 2026 backfilled, 6,318 games). It reproduced production_clv_results.json exactly (n=935, model acc 0.6695 / ll 0.6454 / Brier 0.2249 vs consensus 0.6770 / 0.5771 / 0.1987; REJECT). Resolutions:

• The number is now citable. The prose/artifact mismatch the checker flagged was a stale-prose issue; the artifact was always right and the clean re-run confirms it.
• The "identical folds" are correct OOS design, not a bug. seasons_for_asof for a Feb–Mar 2026 fold returns the two completed prior seasons ['2024','2025']; both are fully in the past, so the game_date < as_of cap correctly removes nothing and every fold trains an identical model. This is a clean out-of-sampleout-of-sampleTested on data that was not used to build or tune the idea. This is the honest test; results on the data you built with are almost always flattering. eval (prior-2-seasons model predicting the held-out 2025-26 odds window, no biasbiasWhether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem." data-def="Accidentally using information that was not available at the time. It makes predictions look brilliant and is the single most common way a backtest fools you.">look-aheadlook-ahead biasAccidentally using information that was not available at the time. It makes predictions look brilliant and is the single most common way a backtest fools you.), not the per-fold refresh the prose overclaimed. Relabel as such; the finding stands.
• Barttorvik drop confirmed safe (predictions bit-identical with the 5 features removed).
• Honest verdict: the production NCAAB model loses to the devigged college market out-of-sample on every metric — worse than the already-rejected stacking model — the expected honest-negative that closes the needs-eval-fix item. Ties nothing.

Real production bugs surfaced (gated follow-ups, offseason — no live harm)

1. NCAAB train_models off-by-one in offseason: default season-window math gives ['2024','2025'] in July 2026, missing the just-completed 2025-26 season (label 2026). Production should train on the 2 most recent completed seasons.
2. NCAAB: drop the 5 inert Barttorvik features + the daily CloudFront-blocked pull (confirmed bit-identical-safe).
3. CFB predict_game history off-by-one: range(season-window, season+1) includes the target season — the leak behind the retrodicted /cfb/performance rows. The walk-forward regen sidesteps it; the live upcoming-predictions path should be fixed directly.