Statistics

The range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all.

The model scored better than the closing line, and the likely range of the gap excludes zero.

How often a model ranks a winner above a loser. 0.5 is a coin flip and 1.0 is perfect. It says nothing about whether the percentages themselves are right.

The average gap between the projection and the actual result, ignoring direction. Lower is better.

Players keep about 20% of their gap from average. 57% stay on the same side.

The price taken was better than the final price before the game. Bettors who beat the close over hundreds of bets tend to profit. It settles about ten times faster than ROI.

The closing line scored better than the model, and the likely range of the gap excludes zero.

Whether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem.

Re-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval.

A score for probability forecasts: the average squared gap between the forecast and what happened. 0 is perfect, 0.25 is a coin flip on even games. Lower is better.

A deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing.

The final odds right before a game starts. It reflects everything the betting market knows, which makes it the hardest benchmark to beat.

Whether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure.

A range of values the true answer is plausibly in. If it includes zero, the effect might genuinely be zero, so we cannot claim it is real.

How closely two things move together, from -1 (opposite) through 0 (unrelated) to +1 (in lockstep). It does not by itself mean one causes the other.

All shot attempts for and against while a player is on the ice: goals, saves, misses and blocks. A stand-in for puck possession.

A gap this size turns up by chance often enough at this sample size that it proves nothing.

How often a quarterback completed passes compared to what was expected given the difficulty of the throws he attempted. It credits accuracy rather than rewarding a QB for only attempting easy throws.

A null result we are confident in, because the same method demonstrably CAN find a real effect when one exists (see calibration). Distinct from simply not having enough data.

Removing the bookmaker's cut from odds to recover the market's actual implied probability.

Next season is a coin flip.

FantasyPros' averaged ranking across many fantasy analysts. A strong benchmark, and hard to beat.

The average profit per dollar if the bet could be repeated endlessly at the fair price. A +3% edge expects to win 3 cents per dollar. Any single bet can lose.

A rating system, originally from chess, that moves a team up or down based on results and the strength of the opponent.

How much a play changed the number of points its team should expect to score on that drive. A better measure of a play than raw yards.

The average result if you could repeat a bet or decision endlessly. Positive expected value means it pays off on average, even though any single instance can lose.

The book's price with its margin removed (exact Shin method), so the two sides add up to 100%.

A method for handling many simultaneous tests, controlling what share of your "discoveries" are expected to be flukes.

Corsi without blocked shots: unblocked shot attempts for and against while a player is on the ice.

A design where each subject is compared against themselves, which cancels out their permanent differences. Here it means comparing a player to his own other seasons rather than to other players.

Injured players weighted by how much they normally play. 1.0 equals one every-down player missing.

Goals a skater added compared with a freely available replacement player. Here it counts offense and penalties only. Defense and ice time are not measured.

Goals a goalie prevented compared with a league-average save percentage on the same number of shots. It ignores shot quality.

Expected goals against minus goals allowed. Positive means the goalie stopped more than an average goalie would have on the same shots.

The share of every dollar bet that the book keeps on average. Also called vig or juice.

Data deliberately set aside and never looked at while developing an idea, then used once at the end as a fair test. Peeking at it first would defeat the purpose.

Measured on the same data used to build or tune the idea. Nearly always looks better than reality.

A formula for how much to stake given your edge. Full Kelly maximizes long-run growth but swings wildly; most people use a fraction of it.

The gap between the model and the closing line is inside the noise.

The range the true value probably falls in, given how much luck is in the sample. If it includes zero, the result could be chance.

A score for probability forecasts that punishes confident wrong answers harshly. Lower is better.

Accidentally using information that was not available at the time. It makes predictions look brilliant and is the single most common way a backtest fools you.

Average size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better.

How far the actual result landed from what the betting market predicted. If this is unrelated to some factor, the market already accounted for that factor.

Consistently moving one direction as the input increases. A real dose-response effect should be monotone; if more of the cause does not mean more of the effect, the pattern is suspect.

Playing out the rest of a season or a game thousands of times with random results drawn from the odds, then counting how often each outcome happens.

The stat does not repeat from one sample to the next. A big number here says little about what comes next.

Testing many ideas at once. Test twenty things and one will usually look significant by luck alone, so results need a stricter bar.

Tested, and nothing was found.

A book's price with its margin removed, so the two sides add up to 100%.

The result is positive, but its likely range includes zero. It could be chance.

The formula the NCAA uses to rank college hockey teams and pick the tournament field. It blends win percentage with strength of schedule.

A test that found nothing. "Null" is the starting assumption that there is no real effect; a "null result" means the data gave us no reason to abandon that assumption. It does not mean the data was missing or the test failed to run.

A test that found nothing real. The starting assumption is "there is no effect here", and a null result means nothing in the data argued against it.

Tested on data that was not used to build or tune the idea. This is the honest test; results on the data you built with are almost always flattering.

The probability of seeing a result this strong purely by chance if nothing real were going on. Small = surprising under "nothing is happening". Below 0.05 is the usual convention, but it is a convention, not proof.

An absolute difference between two percentages. Going from 50% to 53% is 3 percentage points, not 3%.

Randomly shuffling the labels thousands of times to see what results pure chance produces, then checking whether the real result stands out from that.

A standard way to turn an expected count (goals, touchdowns) into the chance of each exact count: 0, 1, 2 and so on.

A fantasy scoring format that awards a point for every catch, which raises the value of high-volume receivers.

The rule and the test were written down and committed before the results were seen, so the test could not be tuned to fit them.

How closely two orderings match. +1 is identical order, 0 is unrelated, −1 is reversed.

Extreme results tend to be followed by more ordinary ones, purely by chance. Mistaking this for a real decline is a classic error.

Players keep about 40% of their gap from average. About 2 in 3 stay on the same side.

Players keep about two-thirds of their gap from average. 3 in 4 stay on the same side.

The typical size of a miss, with big misses counted more heavily than small ones. Lower is better.

Profit as a percentage of the money wagered. +2% means $2 profit per $100 bet.

The rule and the test were written down and committed before the results were seen, so the test could not be tuned to fit them.

The share of the variation in results that a model explains. 1 is perfect. 0 is no better than guessing the average every time. Below 0 is worse than that.

A bootstrap that resamples whole SEASONS rather than individual games, because games within a season are related. Ignoring that makes results look more certain than they are.

When the group being studied was not chosen neutrally, so it differs from the population in ways that distort the answer.

Return relative to how much it bounced around. Higher means smoother returns for the same profit.

A way to remove the book's margin that takes more of it off longshots than favorites. For a two-way market it gives the same answer as subtracting the margin evenly.

How much of the result the projection explains beyond a naive baseline. Above 0 beats the baseline. 0 is no better. Below 0 is worse.

A measure of whether two rankings agree, from -1 (opposite) through 0 (unrelated) to +1 (identical). Cares about order, not exact values.

A repeatability test: split the data in two (here, odd vs even seasons) and check whether the same units score similarly in both halves. If a trait does not repeat, it was noise, not a trait.

The first question: is the effect real at all? Measured against what actually happened, ignoring betting markets.

The second question: is the effect already reflected in the betting odds? An effect can be completely real and still worthless to bet, because the price already includes it.

A measure of how spread out values are. Expressing an effect "per standard deviation" makes different quantities comparable on the same scale.

How much an estimate would move if the same study were run again on new games. Two standard errors either side is roughly the 95% range.

The gap is larger than chance alone usually produces at this sample size.

The share of plays that gained enough to keep the offense on schedule (positive expected points added).

Studying only the things that stuck around. Looking at current players ignores everyone who washed out, which flatters the results.

An effect divided by its own uncertainty. Above about 2 means the effect is unlikely to be chance alone.

The minimum edge needed to overcome the bookmaker's cut. Anything smaller is real but unprofitable.

How far a single result usually lands from the average. About two-thirds of results fall within one swing.

Not enough data to detect an effect even if it is really there. An underpowered null means "we could not tell", which is very different from "there is nothing there".

The model was fitted on earlier seasons and graded on later ones, so it could not memorize the answers. Also called walk-forward or out-of-sample testing.

The bookmaker's built-in cut. It is why a coin-flip bet at -110 needs you to win about 52.4% of the time just to break even.

The gap between the price taken and the final fair price before the game, in percentage points. Positive is better than the close.

Evaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time.

Wins a player added compared with a freely available replacement at the same position.

The chance an average shooter scores from that spot, added up over every shot. It measures chance quality, not finishing.

How far a number sits from average, counted in typical swings. Beyond about 2 is unusual. Beyond 3 is rare.