New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
Do the Injury-Aware Multipliers Actually Improve Projections?
Validation of commit 02fba14, which replaced the projection engine's injury
multiplier table using results from backtests/nfl_context_signals/FINDINGS.md.
Answer: yes, substantially — and the gain comes mostly from fixing the base values, not from the practice-status split that motivated the work.
Why the standard harness can't answer this
scripts/evaluate_fantasy_mae.py states plainly: "MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better. is CONDITIONAL ON
PLAYING: only players the engine projected AND who recorded a stat line that
week are scored." Every injury multiplier is an expected valueexpected valueThe average result if you could repeat a bet or decision endlessly. Positive expected value means it pays off on average, even though any single instance can lose. —
P(plays) × (production | plays) — so most of its mass encodes availability
risk that the conditional frame filters out by construction. Scoring an EV
multiplier only on players who played would make it look worse for the wrong
reason.
Separately, all injury logic sits behind _is_future_week, deliberately, so
the walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. harness sees byte-identical behavior. The shipped change is
invisible to every existing benchmark number. Hence this separate evaluation.
Both frames are reported below, because reporting either alone misleads in opposite directions.
Method
- Out-of-sampleout-of-sampleTested on data that was not used to build or tune the idea. This is the honest test; results on the data you built with are almost always flattering.. Multipliers re-derived on 2015–2021 and evaluated on 2022–2025. The shipped values came from pooled 2015–2025; testing them on that same span would be circular.
- Baseline projection = the player's trailing-4-game mean from strictly prior weeks — the same "naive" baseline the MAE harness uses. This isolates the multiplier's contribution rather than re-evaluating the whole engine.
- Population: skill-position (QB/RB/WR/TE) player-weeks carrying an injury designation. HoldoutholdoutData deliberately set aside and never looked at while developing an idea, then used once at the end as a fair test. Peeking at it first would defeat the purpose. has 2,503 such player-weeks.
- Injury data is point-in-time legitimate: reports are published pre-kickoff,
and the final
date_modifiedrow per player-week is used.
Multiplier stability (develop-only vs shipped pooled)
| Status | Derived on 2015–2021 | Shipped (pooled 2015–2025) |
|---|---|---|
| Questionable | 0.662 | 0.68 |
| — DNP | 0.346 | 0.40 |
| — Limited | 0.689 | 0.69 |
| — Full | 0.867 | 0.87 |
| Doubtful | 0.006 | 0.01 |
| Out | 0.001 | 0.0 |
| Probable | 1.001 | 1.0 |
Re-deriving on a 7-season subset reproduces the shipped table almost exactly. The values are not a pooled-sample artifact.
Results — holdout 2022–2025
Expected-value frame (non-players score 0 — matches the engine's own convention)
| Variant | n | MAE | vs OLD | BiasbiasWhether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem. |
|---|---|---|---|---|
OLD table (pre-02fba14) |
2,503 | 3.414 | — | +2.016 |
| NEW status only | 2,503 | 2.647 | −0.767 | +0.288 |
| NEW + practice split | 2,503 | 2.634 | −0.780 | +0.292 |
−23% MAE, and the bias collapse is the real story: the old table
systematically over-projected injured players by ~2 PPRpoints per receptionA fantasy scoring format that awards a point for every catch, which raises the value of high-volume receivers. points per
player-week. That is exactly what Questionable = 1.0 and Doubtful = 0.25
would predict.
Questionable only — where the change bites hardest
| Variant | n | MAE | vs OLD | Bias |
|---|---|---|---|---|
| OLD table | 1,322 | 6.202 | — | +3.557 |
| NEW status only | 1,322 | 4.996 | −1.206 | +0.535 |
| NEW + practice split | 1,322 | 4.972 | −1.230 | +0.544 |
−20% MAE, bias +3.56 → +0.54. Under the old table a Questionable player was projected at full strength and over-projected by three and a half points.
Conditional-on-playing frame (the standard harness's frame)
| Variant | n | MAE | vs OLD | Bias |
|---|---|---|---|---|
| OLD table | 818 | 5.475 | — | +1.197 |
| NEW status only | 818 | 5.067 | −0.408 | −2.152 |
| NEW + practice split | 818 | 5.200 | −0.275 | −1.965 |
New still wins on MAE here, but now under-projects by ~2 points — correct behavior for an EV multiplier viewed in the wrong frame. Note the practice split is slightly worse than status-only in this frame (5.200 vs 5.067).
Honest caveat: the practice split is the small half
The research finding that motivated this work — practice participation splits
"Questionable" into 41% / 65% / 77% play rates — contributes only
−0.013 MAE beyond fixing the base status values (−0.780 vs −0.767), and it
is marginally negative in the conditional frame. MAE is dominated by getting
the level right, and Questionable = 1.0 was simply wrong by a wide margin.
The practice split still earns its place: it drives play_probability, which is
surfaced in the UI and is the actionable quantity for a start/sit call ("65% to
play"). But it should not be sold as the accuracy win. The accuracy win is
fixing the two broken base values.
Known limitation / possible refinement
The engine now emits an expected value, which is right for "how many points will this player give me." A manager asking "if he suits up, what does he score?" wants the conditional ratio instead (~0.82 / 0.92 / 0.97 relative to healthy for DNP / Limited / Full). Both are computable from the same measurement; only the EV form ships today. Exposing both is a reasonable future addition — the conditional-frame bias of −2.15 above is precisely the cost of using one where the other is wanted.
Reproduce: python3 backtests/fantasy_eval/injury_multiplier_eval.py