New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
Start/Sit Decision Accuracy — 2025
Date: 2026-07-19 · Harness: start_sit_accuracy.py ·
Sample: 64,150 within-(week, position) pairs, 2025 weeks 5–17 · Seed: 20260719
Why pairwise rather than Spearman
SpearmanSpearman correlationA measure of whether two rankings agree, from -1 (opposite) through 0 (unrelated) to +1 (identical). Cares about order, not exact values. answers "is the ordering right on average". A manager asks something narrower: given these two players, which do I start? That is a pairwise decision, so it is graded pairwise — a call is correct when the higher-projected player actually outscores the other. Ties in actual points are dropped.
Benchmarks on the identical pairs: FantasyPros ECRexpert consensus rankingFantasyPros' averaged ranking across many fantasy analysts. A strong benchmark, and hard to beat., and the naive trailing-4-game mean the engine is supposed to beat.
Headline
| Ours | ECR | Naive | |
|---|---|---|---|
| All 64,150 pairs | 69.5% [68.7, 70.4] | 71.5% [70.6, 72.5] | 69.1% [68.2, 70.1] |
69.5% sounds strong, and it is close to meaningless on its own — most pairs are blowouts nobody deliberates over. The number that describes decision quality is the close band.
By projected-points gap — where the aggregate falls apart
| Gap | Ours | ECR | Naive | n |
|---|---|---|---|---|
| coin-flip (<2 pts) | 54.6% [53.8, 55.2] | 60.7% [59.2, 62.4] | 54.2% [52.6, 56.0] | 14,321 |
| close (2–5) | 63.8% [63.1, 64.5] | 66.1% [65.5, 67.1] | 62.8% [61.9, 63.9] | 18,247 |
| clear (5–10) | 75.2% [74.1, 76.4] | 75.5% [74.6, 76.5] | 75.3% [74.3, 76.4] | 20,405 |
| blowout (10+) | 87.4% [85.2, 89.5] | 86.8% [84.7, 88.9] | 87.3% [85.1, 89.5] | 11,177 |
Three things fall out, and they matter more than the headline:
- On the decisions that actually get deliberated we are barely better than a coin flip. 54.6% in the <2-point band — and that band is 22.3% of all pairs. The projection is close to uninformative exactly where a manager wants help.
- The naive trailing mean is statistically indistinguishable from the engine everywhere. 69.1% vs 69.5% overall; 54.2% vs 54.6% on close calls. Every interval overlaps. All the engine's machinery buys ~0.4pp of decision accuracy over a 4-game average.
- The engine's apparent competence is borrowed from easy pairs. At gaps of 5+ points, ours, ECR and naive are all within noise of each other (75%, 87%) — those calls are decided by who the players are, not by any model.
Where ECR's edge actually lives
ECR beats us by 2.0pp overall, but that is concentrated almost entirely in the close band: +6.1pp on coin-flip pairs (60.7% vs 54.6%), +2.3pp on close, and nothing on clear or blowout pairs (−0.3pp and −0.6pp, both inside noise).
This sharpens the earlier ECR finding considerably. The expert edge is not general forecasting superiority — on the easy calls they are no better than a trailing average. Their advantage is concentrated in exactly the deliberated decisions, which is what late-week injury, practice-participation and role information should buy, and is consistent with the snap/depth result closing ~30% of the ranking gap.
By position
| Pos | Ours | ECR | Naive |
|---|---|---|---|
| QB | 61.9% | 63.2% | 61.0% |
| RB | 71.6% | 74.1% | 72.0% |
| TE | 70.1% | 71.7% | 68.8% |
| WR | 69.0% | 70.8% | 68.6% |
RB is the one position where the naive mean beats the engine (72.0% vs 71.6%), consistent with the MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better. report independently flagging RB volume as the weakest link.
Honest answer to "how good is it at start/sit?"
Good at easy calls, near-useless at hard ones. On a genuine 50/50 the engine is 54.6% — about 1 extra correct call per 22 deliberated decisions versus flipping a coin, and no better than a 4-game average. A human using this for start/sit is getting most of the value from knowing who the good players are, not from the projection.
The actionable read: the close band is the entire product problem. It is where ECR wins, where role information helped ranking, and where the engine currently adds nothing over naive. Anything that does not move the <2-point band is not improving start/sit, whatever it does to aggregate MAE.