Tennis Elo Backtest Reconciliation — 2026-07-12

Why: the live tennis_elo_v1 scanner's DEFAULT_MIN_EDGE=5% gate traced to a deleted script (cleanup 7ef5f27) whose +6.25% WTA-grass claim was contradicted by backtests/stacking_eval/tennis. Methodology review action item 3.

Method: ultracode chain — T1 recover+rerun the deleted backtest; T2 reconcile the contradiction by single-switch isolation experiments; T3 test the dispersion hypothesis with an untouched 2022-2024 holdoutholdoutData deliberately set aside and never looked at while developing an idea, then used once at the end as a fair test. Peeking at it first would defeat the purpose.. Two adversarial checks (one flagged a provenance flaw in a supporting CI — noted below; core mechanisms verified by rerun).

T1 — recovered backtest, rerun on current data

Test population: 77,471 matches with usable Pinnacle closing odds (2010-01-03 to ~2025, source IS NULLnull resultA test that found nothing. "Null" is the starting assumption that there is no real effect; a "null result" means the data gave us no reason to abandon that assumption. It does not mean the data was missing or the test failed to run. confirmed identical to unfiltered).

OVERALL (all tours/surfaces, unwarmed), ROIreturn on investmentProfit as a percentage of the money wagered. +2% means $2 profit per $100 bet. by edge threshold: edge≥1%: n=72455 ROI=-6.09% | edge≥2%: n=67529 ROI=-6.34% | edge≥3%: n=62739 ROI=-6.63% | edge≥5%: n=53429 ROI=-6.96% | edge≥7%: n=45034 ROI=-7.34% | edge≥10%: n=34191 ROI=-7.96%

OVERALL (warmed, both players ≥10 prior matches on surface): edge≥1%: n=50585 ROI=-5.05% | edge≥2%: n=46509 ROI=-5.28% | edge≥3%: n=42558 ROI=-5.59% | edge≥5%: n=34989 ROI=-5.80% | edge≥7%: n=28322 ROI=-5.87% | edge≥10%: n=20036 ROI=-5.71%

BY TOUR×SURFACE (warmed), edge≥5%: atp/Clay n=5626 ROI=-8.60% | atp/Grass n=1507 ROI=-0.32% | atp/Hard n=10873 ROI=-7.52% | wta/Clay n=4616 ROI=-6.02% | wta/Grass n=1258 ROI=+7.46% | wta/Hard n=11109 ROI=-4.85%

WTA/Grass is the only segment consistently positive across thresholds (edge≥1%: +5.64% ... edge≥10%: +8.94%), but year-by-year 2011-2025 swings from -31.16% (2017) to +70.64% (2014) — no stable signal, single-year noise dominates.

Log-loss (mean -log p on actual winner, lower=better): Overall (n=77471): ELOElo ratingA rating system, originally from chess, that moves a team up or down based on results and the strength of the opponent.=0.6343 vs Market=0.5832 (market beats ELO by 0.0511) WTA/Grass (n=4196): ELO=0.6609 vs Market=0.5974 (market beats ELO by 0.0635) WTA/Grass warmed (n=1698): ELO=0.6663 vs Market=0.6288 (market beats ELO by 0.0376) Market beats the ELO model on every slice tested — no calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing. edge anywhere.

Headline reconciliation — WTA/Grass ROI at edge≥5% (warmed, n=1258, confirmed exactly 1258 unique match_ids = one bet per match, so match-clustering the bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval. = standard iid bootstrap over bet rows, no extra correction needed unlike per-book NHL rows): Point ROI = +7.46%. Match-clustered bootstrap 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. (10,000 resamples, numpy default_rng seed=20260712): [-1.89%, +17.23%]. CI includes 0 — not statistically significant.

Calibration table (predicted vs actual win rate, all 77471 rows): well-calibrated overall, gaps ≤3.5pp at the extremes (0.05 bucket: predicted 0.05 vs actual 8.5%; 0.95 bucket: predicted 0.95 vs actual 91.5%), consistent with a reasonably calibrated but not market-beating model.

T2 — reconciliation verdict

WHICH IS CORRECT: Neither is "wrong," and read fully (not just headlined) they don't actually disagree: they are non-comparable numbers being described as if comparable. Exhibit A's own report already flags its headline WTA/Grass +7.46% as CI-crossing-zero and year-by-year noise-dominated (-31% to +71%). Exhibit B's REPORT.md never computes a WTA/Grass-specific ROI at all — its "ROI negative across the board" table is for a different object (its consensus-stacked model, unwarmed, dual-sided EV%). I reproduced Exhibit B's own bet-structure (EV%, both sides, executed at Pinnacle) using RAW Elo instead of the stacked blend, correctly warm-filtered: WTA/Grass comes out +3.5% to +4.1% ROI — same sign and same order of magnitude as Exhibit A, and, like Exhibit A's number, not statistically significant (bootstrap 95% CI [-4.9%,+13.0%] at min_ev=5%). So under match-clustered bootstrapping, BOTH methodologies agree: no significant WTA/Grass edge exists. The apparent contradiction was an artifact of comparing Exhibit A's warmed/Pinnacle/raw-Elo number to Exhibit B's unwarmed/avg-anchored-stacked-model number, not a real disagreement about the world.

ROOT CAUSE: Four compounding methodological differences, isolated one-at-a-time by holding Exhibit A's bet structure fixed and flipping single switches (script: backtests/tennis_eval/reconcile_experiments.py, read-only against [redacted], seed=20260712 for all bootstraps):

1. PRICE SOURCE (dominant, confirmed prime suspect): Exhibit A devigs+executes at Pinnacle close (ps_w/ps_l). Swapping only the price source to the market-average (avg_w/avg_l) that Exhibit B's "consensus" feature uses flips WTA/Grass ROI from +7.46% to -1.53% (warmed) / +6.41% to -2.40% (warmed+Completed-only) — sign flip, single largest lever.

2. WARM-UP / COLD-START FILTER: Exhibit A requires both players ≥10 prior matches on surface for its headline number; Exhibit B's ROI backtest applies no such filter at all. Running Exhibit B's own EV%/dual-side/Pinnacle bet structure on raw Elo WITHOUT a warm filter gives WTA/Grass ROI of -2.7% to -3.1% (matches B's "negative across the board" framing); adding the identical ≥10-match warm filter flips it to +3.5% to +4.1% (matches A's sign/magnitude). Second-largest lever, and it alone reproduces B's negative headline from an otherwise-identical-to-A signal.

3. STACKING-IN-CONSENSUS: Exhibit B's reported numbers (REPORT.md "raw Elo ties/loses to consensus," the ROI table) are actually for a logistic-regression blend whose dominant, most-weighted feature IS the devigged market probability itself — not raw Elo alone. run_experiment.py never isolates raw elo_p1_prob for its accuracy/log-loss/ROI comparisons; I ran that isolation directly (Experiment 4) and it does independently confirm raw Elo loses to consensus on calibration (log-loss 0.6555 vs 0.5975 on WTA/Grass OOS window) — that part of B's claim holds up. But for ROI specifically, even warmed+Pinnacle, the stacked model's WTA/Grass ROI (-1.4% to +0.6% across thresholds) is materially worse than raw Elo's own (+3.5% to +4.1%) run through the identical bet structure — because a model anchored to market consensus by construction can't diverge from the market enough to generate the same apparent edge it's being tested against. This is a real, independent third driver, separate from #1 and #2.

4. Retirement/'Completed'-only filter: tested directly and found negligible (+7.46%→+6.41%, no sign change) — ruled OUT as a driver, despite being a real population difference between the two scripts.

5. Bet structure (A: one bet/match on sign(prob-edge) vs B: independent EV%-threshold check on both sides): secondary — changes magnitude (~+7% vs ~+4%) but never flips sign on its own; verified zero duplicate-side bets per match in practice (dual-sided B structure collapses to one-bet-per-match empirically on this data).

Underneath all four: once match-clustered-bootstrapped (10,000 resamples, seed=20260712), no reconciled WTA/Grass configuration — Exhibit A's own (n=1258, CI [-1.89%,+17.23%]) or Exhibit B's structure run on raw Elo (n=1343, CI [-4.89%,+13.02%]) — excludes zero. Exhibit A's own year-by-year table (already in Exhibit A, -31.16% in 2017 to +70.64% in 2014) independently confirms single-season noise, not a stable signal.

EDGE GATE VERDICT: DEFAULT_MIN_EDGE=5% should not be used as a live betting gate for tennis raw-Elo at all, at any value — this is a "no edge exists" finding, not a "raise the bar" fix. Evidence: (a) Exhibit A's own OVERALL (non-niche) numbers stay negative at every threshold from 1% to 10% (warmed: -5.05% at edge≥1% down to -5.71% at edge≥10%) — no threshold rescues the aggregate. (b) The one niche that ever looked positive (WTA/Grass, warmed, Pinnacle-priced) tops out at a statistically insignificant point estimate under BOTH reconciled methodologies (+7.46% CI[-1.89,+17.23] and +3.5-4.1% CI[-4.9,+13.0]) — raising or lowering the 5% threshold within that niche doesn't change the significance verdict (checked at 1/2/3/5% min_ev, all CIs cross zero). (c) The signal is provably fragile to reasonable, defensible methodology choices (price source, warm-filter, stacking) that a live scanner must also make — a gate tuned to survive one specific choice-set isn't robust. Recommendation: keep tennis Elo analytics/informational-only (consistent with the existing memory note "tennis serve-Elo rejected... no alpha"); if a scanner config literally has a tennis-specific DEFAULT_MIN_EDGE, disable it as a betting trigger rather than retuning the number — do not deploy any edge≥X% tennis-Elo-vs-price betting rule off this backtest evidence.

NUMBERS: All reproduced in backtests/tennis_eval/reconcile_experiments.py output (read-only, seed=20260712, 10k-resample match-clustered bootstrap):

Exp 1 (Exhibit A structure, WTA/Grass, edge≥5%, warmed unless noted): - A exact (Pinnacle, incl. retirements): n=1258, ROI=+7.46% - +Completed-only filter: n=1223, ROI=+6.41% (negligible effect) - +avg price instead of Pinnacle: n=1286, ROI=-1.53% (SIGN FLIP) - +Completed-only AND avg price: n=1246, ROI=-2.40% - -warmed filter removed: n=3350, ROI=-0.50% (SIGN FLIP toward zero/negative)

Exp 2 (bootstrap 95% CI on the above): none exclude zero — A exact CI=[-1.89%,+17.23%]; +Completed-only CI=[-3.18%,+16.17%]; +avg price CI=[-9.96%,+7.14%]; both switches CI=[-11.03%,+6.31%]; unwarmed CI=[-6.23%,+5.33%].

Exp 3 (Exhibit B structure — EV%, both sides, Pinnacle execution — raw Elo vs Exhibit B's actual stacked model, WTA/Grass): - Unwarmed: raw-Elo ROI -2.73% to -3.10% across min_ev 1-5%; stacked ROI -6.70% to -10.02% (both negative, matches B's reported framing). - Warmed (≥10 prior matches both players): raw-Elo ROI +3.53% to +4.08% (n=1343-1488); stacked ROI -2.21% to +0.61% (n=140-632) — stacking suppresses/kills the same edge raw Elo shows. - Bootstrap CI on raw-Elo-warmed: min_ev=5%: n=1343, point=+4.03%, CI=[-4.89%,+13.02%] — not significant.

Exp 4 (raw Elo vs devigged consensus, calibration only, no stacking): WTA/Grass OOS (n=3562, years≥2013): elo_ll=0.6555 vs cons_ll=0.5975, elo_acc=0.6151 vs cons_acc=0.6743 — market beats Elo on calibration, consistent with Exhibit A's own calibration table (elo_ll 0.6609/0.6663 vs cons 0.5974/0.6288) — this part of both reports agrees and was never actually in dispute.

Underlying population/DB facts confirmed: Elo formula identical in both exhibits (1/(1+10^(-diff/400)), K=24, from scripts/tennis/build_elo.py); only literal 'Walkover' rows excluded upstream (retirements included) — Exhibit A inherits that, Exhibit B additionally filters comment='Completed'; source IS NULL population = 81,185 rows joined (77,849 with Pinnacle odds, 79,295 with avg odds, 78,660 Completed).

T3 — dispersion hypothesis (soft-book vs Pinnacle), untouched 2022-24 holdout

Built backtests/tennis_eval/dispersion_test.py (read-only against [redacted], no commits made). Population: tennis_match_elo JOIN tennis_matches, source IS NULL, 2013-01-01..2024-12-31 (58,033 rows). Selection window 2013-2021 used only to look at the threshold sweep; 2022-2024 held out and never touched for threshold selection. Only two columns in tennis_matches represent a literal single tradeable bookmaker quote: ps (Pinnacle, the sharp anchor already used throughout this codebase) and b365 (Bet365, retail/soft) — max and avg are synthetic panel stats, not a counterparty you can bet at, so b365 is used as both "that book" in rule (a) and "the soft book" in rule (b), keeping the two rules apples-to-apples on the identical book. Coverage report by year for all 4 price columns is printed first; fill rate is 99-100% for ps/b365 every year, but 2020 is flagged as thin on match VOLUME (2,309 matches vs ~5,100 median, COVID-shortened season) — not pooled silently, called out explicitly in output.

Rule (a): edge = elo_p_w - devigdevigRemoving the bookmaker's cut from odds to recover the market's actual implied probability.(b365)_p_w, bet the side Elo favors when |edge| >= threshold, settled @ b365. Rule (b): disp_w = devig(ps)_p_w - devig(b365)_p_w (positive = soft book underrates w vs Pinnacle = value on w), bet triggered when |disp_w| >= threshold AND Elo agrees on which side (elo_p_side > 0.5) — Elo only confirms direction, doesn't set magnitude, per the task's dispersion-rule spec. Every bet is settled two ways off the identical match/side selection: ROI @ book (tradeable) and ROI @ Pinnacle close (CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure. diagnostic, not tradeable) — a rule whose book-ROI holds but whose Pinnacle-priced ROI collapses is exposed as stale-price artifact rather than durable skill. One bet per match throughout (never >1 side/match), so bootstrap resampling of the bet array is already match-clustered per the house CI rule; 10,000-resample bootstrap, seed=20260712, chunked to bound memory (initial unchunked version OOM-killed on the ~27k-bet overall-population arrays).

RESULT: (b) does NOT show positive holdout ROI where (a) does not. Both rules fail to clear the "significant positive holdout ROI" bar at every threshold from 2%-8%, on both the overall population and the WTA/Grass supplementary slice — a valid "neither survives" answer per the task's own framing. More than that, rule (a) shows a STRONG, STATISTICALLY SIGNIFICANT NEGATIVE holdout ROI at every single threshold (2-8%), at both book and Pinnacle pricing, on the overall population (e.g. edge>=5%: n=6,937, ROI@book=-9.44% CI=[-12.72%,-6.09%]) — the "current rule," when swept broadly rather than restricted to the one hand-picked WTA/Grass niche, is actively anti-predictive, consistent with the standing methodology-review finding that Elo loses to market calibration. Rule (b)'s overall-population bet counts collapse fast as threshold rises (n=5,607 at thr=2% down to n=1 at thr=8%) and no CI ever excludes zero on the holdout. On WTA/Grass specifically, rule (b) at thr=2% is actually significantly NEGATIVE in both windows (selection CI=[-21.29%,-0.48%], holdout CI=[-38.79%,-2.56%]), and at higher thresholds the WTA/Grass holdout n drops to single digits or zero (uninterpretable, correctly reported as such rather than hidden).

Coverage (source IS NULL, 2013-2024): ps 99-100% fill every year; b365 99-100% fill every year; max/avg 100% every year; 2020 flagged thin on VOLUME only (2,309 vs median 5,096).

OVERALL POPULATION, Rule (a) [Elo-devig(b365) edge, settled@b365], warmed (>=10 prior matches/surface/side): thr=2%: select n=26,845 ROI@book=-8.62% CI=[-10.21,-6.97]; HOLDOUT n=9,466 ROI@book=-9.26% CI=[-11.88,-6.50], ROI@pin=-6.29% CI=[-9.06,-3.36] thr=5%: select n=19,955 ROI@book=-9.25% CI=[-11.11,-7.26]; HOLDOUT n=6,937 ROI@book=-9.44% CI=[-12.72,-6.09], ROI@pin=-6.40% CI=[-9.85,-2.89] thr=8%: select n=14,213 ROI@book=-8.88% CI=[-11.24,-6.52]; HOLDOUT n=4,922 ROI@book=-10.69% CI=[-14.65,-6.64], ROI@pin=-7.72% CI=[-11.85,-3.42] -> negative and CI-excludes-zero at EVERY threshold, EVERY window, EVERY pricing.

OVERALL POPULATION, Rule (b) [devig(b365)-devig(ps) dispersion, Elo-confirmed, settled@b365]: thr=2%: select n=5,607 ROI@book=-0.96% CI=[-2.92,+0.98]; HOLDOUT n=1,795 ROI@book=-2.76% CI=[-6.21,+0.75] thr=4%: select n=897 ROI@book=+4.37% CI=[-0.64,+9.58]; HOLDOUT n=190 ROI@book=+0.57% CI=[-10.14,+11.26] thr=5%: select n=291 ROI@book=+8.88% CI=[-0.66,+18.39]; HOLDOUT n=46 ROI@book=-2.41% CI=[-26.76,+21.72] thr=8%: HOLDOUT n=1 (uninterpretable) -> no threshold gives a holdout CI excluding zero.

WTA/GRASS SUPPLEMENTARY, Rule (a): thr=5% select n=729 ROI@book=-5.56% CI=[-17.02,+6.04]; HOLDOUT n=307 ROI@book=+1.67% CI=[-14.74,+18.51] — point estimate flips positive in holdout but CI crosses zero at every threshold 2-8%. WTA/GRASS SUPPLEMENTARY, Rule (b): thr=2% select n=176 ROI@book=-10.69% CI=[-21.29,-0.48]; HOLDOUT n=68 ROI@book=-20.76% CI=[-38.79,-2.56] (significantly NEGATIVE, both windows); thr>=6% holdout n<=5 or 0 (too thin to interpret, flagged as such rather than pooled).

Full threshold-by-threshold table (2-8%, both windows, both pricings, both rules, overall + WTA/Grass) is in the script's stdout.

Adversarial check notes

• Check 1 (reconciliation validity): flawed — one supporting bootstrap CI in T2 (raw-Elo warmed WTA-grass min_ev=5%: +4.03% CI [-4.89, +13.02]) was attributed to reconcile_experiments.py which does not compute it. The isolation experiments and sign-flip mechanisms were independently rerun and verified. Treat that one CI as unsourced; the conclusion does not rest on it (T1's directly-computed CI [-1.89, +17.23] and T3's holdout tables carry the result).
• Check 2 (holdout integrity + statistics): sound.

Bottom line

Three independent walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. evaluations now agree: no supported tennis Elo edge exists in any tested framing. Overall Elo-edge ROI is significantly NEGATIVE at every threshold (holdout CIs exclude zero). The only ever-positive slice (WTA grass, warmed, Pinnacle-priced) is statistically insignificant under both methodologies. The dispersion reframing also fails its holdout (and is significantly negative on WTA grass). The original +6.25% claim arose from the luckiest corner of (price source x warm-filter x niche) — a selection artifact, not an edge. The scanner's premise is unsupported; its live CLV record remains the only forward judge, and expectations should be that it confirms this null.