Snap Counts + Depth Charts Close ~30% of the ECR Gap

Date: 2026-07-19 · Season: 2025, weeks 7–17 · Harness: snap_depth_experiment.py → snap_depth_experiment_results.json · Seed: 20260719

Why this test

The 2025 walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. comparison put our projections −0.0674 SpearmanSpearman correlationA measure of whether two rankings agree, from -1 (opposite) through 0 (unrelated) to +1 (identical). Cares about order, not exact values. behind FantasyPros ECRexpert consensus rankingFantasyPros' averaged ranking across many fantasy analysts. A strong benchmark, and hard to beat., and the deficit was uniform across QB/RB/TE/WR rather than concentrated in any position. That shape argues for missing information rather than a broken position model — and the cheapest candidate is role: how many snaps a player is actually getting, and where he sits on the depth chart. Experts see that. A projection built from completed box scores partly does not.

Both inputs are free in nflreadpy and already local, so this was testable in hours with no credentials and no ToS question — unlike the Reddit sentiment alternative, which is blocked (403 + Disallow: /).

Pre-registered before running

• Primary: the augmented ranking beats our own baseline projection ranking, week-clustered 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. on the paired difference excluding zero.
• Secondary: it narrows the gap to ECR.
• A nullnull resultA test that found nothing. "Null" is the starting assumption that there is no real effect; a "null result" means the data gave us no reason to abandon that assumption. It does not mean the data was missing or the test failed to run. was an acceptable outcome: it would have said role information is already priced into the projection.

Feature set and ridge λ=1.0 were fixed once, a priori — no grid search, so no threshold snooping.

Leakage discipline

Every feature for week W uses only data strictly before W: - snap features are shift(1)-then-rolled within player (trailing 3-game snap share, 1-week trend, games-with-snaps count) - the depth chart is the latest snapshot whose timestamp precedes the first kickoff of week W - the blending ridge is refit each week on weeks < W only

Weeks 5–6 are unscored because fewer than 250 prior training rows exist.

Result — PRIMARY PASSES

44 cells, 2,097 player-weeks.

Secondary — the ECR gap narrows by ~30%

On the 18 cells with complete ECR coverage:

ECR still wins — both intervals exclude zero — but roughly 30% of the gap closes on role information alone.

By position

All four improve. QB gains most — and QB was also the largest ECR deficit in the 2025 comparison (−0.0881). Information helping most exactly where the gap was widest is coherent rather than coincidental, and it is the first evidence distinguishing "we lack information" from "our model is worse."

Honest bounds

1. The effect is small. +0.0185 Spearman is real but modest; it does not make us competitive with expert consensus, it makes us less far behind.
2. The ECR arm is thin. Only 18 of 44 cells have complete ECR coverage because a cell is dropped if any player in it lacks an ECR rank. That is deliberately conservative, and the baseline-vs-ECR figure it produces (−0.0548) is consistent with the headline −0.0674 from the full comparison.
3. One season, 11 scored weeks. Enough for the primary CI to exclude zero; not enough for confident per-position claims.
4. Depth-chart pos_rank is a coarse proxy; missing values default to 4.

What this implies

The "information, not modeling" hypothesis survives its first real test. The follow-up that actually matters is whether the remaining −0.0386 is reachable at all — the obvious residual candidates (practice participation, injury designations, beat-reporter intel) are progressively harder to get and are exactly what a 100-expert consensus aggregates.

Productionizing this is a separate decision: these features improve ranking, and the engine's headline metric is per-stat MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better.. Wiring snap share into the projection engine itself (rather than as a post-hoc re-ranker) should be judged on MAE before it ships.

Follow-up: wiring snap share into the engine FAILS the MAE gate

The re-ranking result above does not survive translation into the projection engine. Snap share was added as snap_share_weight (volume stats only — snaps buy opportunity, not efficiency), with the trend measured strictly on prior weeks, and A/B'd on the identical 2025 weeks 5–17 walk-forward evaluation.

1 improved, 3 worsened. Every non-volume category shows exactly +0.000, confirming the feature was correctly scoped rather than leaking into efficiency.

The feature did fire. Reproducing the multiplier over 9,681 player-weeks: mean 1.0016, sd 0.0965, 47.8% of players adjusted by >5%, 32.8% by >10%, 18.7% pinned at the clamp. This is a genuine negative, not a no-op that never engaged.

Why both results can be true

They measure different things, and the distinction is the point:

• Ranking (Spearman) rewards getting the relative order right. A snap-trend multiplier that lifts players gaining a role and drops those losing one improves ordering even if its magnitude is wrong.
• MAE is about the absolute level. A multiplier with the right sign but a miscalibrated size adds variance without reducing biasbiasWhether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem., so it can help ordering and hurt (or not move) point accuracy.

No weight search was run. 0.5 was fixed a priori and it failed; tuning downward until the number turns green is exactly the threshold snooping this platform refuses elsewhere, and a weight small enough to "pass" is just a slower way of shipping 0.0.

Disposition

snap_share_weight ships at 0.0 (off). The code stays, documented and A/B-able, so the next attempt starts from a measured baseline instead of re-deriving it. If snap information is to help the engine, it likely has to enter as a volume model input rather than a post-hoc multiplier on a trailing mean — the multiplier form is what failed here, not necessarily the signal.

Second attempt: snap share as a VOLUME MODEL INPUT — also fails, and worse

The multiplier form failing left open whether the signal was useless or just the form. So the structurally different version was tried: decompose each volume stat into (per-snap rate) × (projected snaps), estimating the rate over the long decayed window (a stable skill) and the snaps over a short one (role changes fast). A back going from 30% to 70% of snaps then picks up volume immediately rather than waiting for a 16-game mean.

0 improved, 3 worsened — and worse than the multiplier it was meant to improve on. All non-volume categories again exactly +0.000.

Why passing_attempts moved exactly 0.000

Not a bug. Starting QBs play ~100% of snaps (spot-checked: Mahomes 1.0/1.0/1.0, Allen 1.0/0.96/1.0 over their last three), so rate × snaps reduces to attempts/1.0 × 1.0 — algebraically the trailing mean. The decomposition can only act where snap share varies.

(Correction worth recording: the first check of this used snap share across all QB rows and found sd = 0.341, which appeared to refute the explanation. That population includes backups who barely appear in the projection sample. On the QBs actually scored, share really is ~1.0. The right population matters.)

Why the decomposition is worse, not just neutral

Bias–variance, and it goes the wrong way. The trailing mean is a biased but low-variance estimator: it lags a role change, but it averages 16 games of one quantity. Rate × snaps trades that for two noisy estimates multiplied together — a per-snap rate whose denominator is small enough to amplify noise, times a 3-game snap average. The lag it removes costs less than the variance it adds. On the positions where snap share genuinely varies (RB/WR/TE) it is strictly worse, and RB — the highest-variance role — degrades most (+0.045).

Overall disposition

Two structurally different attempts, both pre-registered, both failed:

Both ship off. The honest conclusion is now stronger than after the first failure: snap share improves ranking (+0.0185 Spearman, CI excludes zero) and does not improve point accuracy in either the multiplicative or the decomposed form. Those are different objectives and this signal only serves one of them.

The practical read: if the fantasy surface is used for start/sit and lineup ORDER, a post-hoc snap re-ranker is worth shipping on that surface. It should not be pushed into the projection engine, whose headline metric is MAE.