Our Fantasy Projections vs FantasyPros Expert Consensus (ECR)

2025 rerun (2026-07-19) is the headline result — verified walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. on our side, 98% coverage, all four positions individually significant. The 2024 section below is retained as the first pass and for its provenance caveat, but where the two disagree, trust 2025.

2025 — walk-forward, weeks 5–17 (PRIMARY)

Our side is the strict walk-forward regeneration from scripts/evaluate_fantasy_mae.py --dump-projections (2,438 player-weeks): the engine is fed one historical week at a time and may only use data from before the week it projects. Unlike the 2024 db rows, this is provably date-capped, so the provenance asterisk does not apply.

Coverage: 2,394 of 2,438 matched (98.2%; only 44 unmatched by name, 0 weeks missing). We rank better in 8 of 52 cells.

What changed versus the 2024 pass

1. The loss is confirmed and the estimate tightened — −0.0674 with a much narrower CI, on verified walk-forward output and near-complete coverage.
2. The position story did NOT replicate. In 2024 the loss looked concentrated in RB and WR, which tempted a tidy narrative about RB volume matching the engine's own MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better. weak spot. In 2025 the deficit is uniform — all four positions individually significant, with QB now the largest gap. The 2024 concentration was a one-season artifact and that interpretation is withdrawn. The robust finding is a broad ~0.04–0.09 ordering deficit, not a position-specific hole.
3. Both sides score higher in absolute terms (ours 0.477 vs 0.324) on a larger, more complete matched sample — so the levels are not comparable across the two sections, only the gaps.

2024 — first pass (secondary; see provenance caveat)

Date: 2026-07-19 · Season: 2024 · Harness: compare_vs_ecr.py → compare_vs_ecr_results.json · Seed: 20260719

Question

Does a crowdsourced expert consensus rank fantasy players better than our engine does, on the same player-weeks? This is the first time the platform's fantasy projections have been scored against any outside forecaster.

Method

ECR is a rank, not a point projection, and no free source publishes FantasyPros' projected points — so the only apples-to-apples comparison is ordering. Both sides are converted to a within-(week, position) ranking and scored by Spearman correlationcorrelationHow closely two things move together, from -1 (opposite) through 0 (unrelated) to +1 (in lockstep). It does not by itself mean one causes the other. against actual PPR points. Ordering is also the metric that matters operationally: start/sit and lineup order are ranking decisions, not point-estimate decisions.

• Only player-weeks present on both sides, same position, same week, with a settled actual. ECR's 2024 archive begins at week 4, so weeks 1–3 are dropped rather than counted as a default win for us.
• Cells with fewer than 6 players are dropped (a Spearman over 3 tight ends is noise).
• BootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval. clustered by week (2,000 reps): players inside a week share opponents, weather and game environments, so iid resampling would understate the interval.

Result — we lose, and the gap excludes zero

We rank better in only 13 of 56 (week, position) cells — 23%.

By position

The overall loss is driven by RB and WR, the two highest-volume positions. QB and TE are not statistically distinguishable. This is consistent with the engine's own MAE report, which independently flags RB volume as the weakest link (committee churn) — the same failure mode surfacing under a different metric and against a different benchmark.

Caveats that materially bound this

1. Provenance asymmetry — the important one. ECR is a genuine real-time artifact: a Friday scrape of what experts published before kickoff, so hindsight is impossible. Our side is not equivalent — every 2024 row in fantasy_projections carries a single timestamp (2025-12-26T19:48..20:00), meaning the season was regenerated in one ~12-minute batch fourteen months after it ended. Whether that regeneration was date-capped per week is not established by this harness.

This cuts both ways and should not be smoothed over: - If the regeneration was properly walk-forward, this is a fair loss. - If it leaked, our side was flattered and still lost, making the result worse for us, not better.

The observed correlation (~0.32) is nowhere near the ~0.9 a grossly leaked projection would produce, so no gross leak is evident — but "no gross leak" is not "verified walk-forward". Treat the direction as solid and the magnitude as indicative.

2. Top-of-roster only. FantasyPros ranks only the fantasy-relevant top of each position. The matched sample averages 10.43 projected PPR; the 841 unmatched rows average 6.90. This says nothing about deep-bench ordering — which is where our engine projects the most players.

3. One season, 14 weeks. 56 cells. Enough for the aggregate CI to exclude zero, not enough for confident per-position claims (QB rests on 164 player-weeks).

Interpretation

Their edge is almost certainly information, not modeling: beat-reporter intel, practice participation, snap-count trends and injury designations, all aggregated across ~100 experts. Our engine sees only completed box scores. None of that is recoverable by a better regression.

This also sits alongside the engine's other honest negative — scored against real sportsbook prop closes it ties on Brier (0.1201 vs 0.1151) and loses badly on log losslog lossA score for probability forecasts that punishes confident wrong answers harshly. Lower is better. (0.5216 vs 0.3808). So the engine loses to the market on calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing. and to the expert consensus on ordering. It beats a naive trailing 4-game mean by ~3% on MAE, and that remains the honest summary of its skill.

What would move it

1. Fix the provenance gap first. Archive projections at generation time with a real timestamp so a future comparison is against genuine pregame output. Until then every fantasy evaluation carries this asterisk.
2. Attack RB volume, the position where the loss is largest and significant, and which the MAE report independently flags.
3. Forward-capture ECR (already running, capture_ff_ecr.py, Saturdays) so 2026 can be scored on genuine real-time output from both sides.

QB/ECR blend — does NOT replicate (2026-08-13)

A blend was floated in conversation after the 2025 primary result above ("QB w*=0.55 gives +0.0349, t=3.10"), but it was never written to a script or a results file — only stated in chat. Before wiring anything into [redacted] on the strength of an unlogged number, qb_ecr_blend.py reproduced it from scratch on the same 2025 walk-forward data this doc's primary result uses.

It does not replicate. The opposite pattern holds. Blending our QB projection with ECR in rank space (blended = w·our_rank + (1-w)·ecr_rank) degrades monotonically as w increases — there is no interior optimum near 0.55 or anywhere else. Pure ECR (w=0) is the best point on the entire sweep:

w=0.50 vs pure ECR: −0.0344, 95% CI [−0.0682, −0.0005] — excludes zero, in the direction opposite the earlier claim. The argmax over the full grid is w=0, and even that isn't distinguishable from w=0.50 (CI spans zero) — there's no real interior structure here at all, just "ECR alone, or close to it, beats every blend."

The (non-walk-forward, provenance-uncertain) 2024 batch-regenerated data was checked too, since that's the only other dataset available. It shows the opposite shape again — mildly monotonemonotoneConsistently moving one direction as the input increases. A real dose-response effect should be monotone; if more of the cause does not mean more of the effect, the pattern is suspect. increasing toward w=1 (pure engine), peaking at +0.4585 vs +0.4436 at w=0 — which also does not support an interior peak near 0.55. Two different datasets, two different (non-)shapes, neither matching the original claim.

Conclusion: no QB blend was implemented. The premise that motivated it doesn't hold under the primary evaluation this whole document is built on. If anything, this is a second, independent confirmation that ECR's QB edge (the largest gap of any position in the primary result) isn't partially recoverable by a simple linear blend — it would take an actual different signal, not a weighted average of the two existing ones.