We tried to break our own metrics

Three rules apply to everything below. Nothing is graded on data it was fit on. Nothing uses information that would not have existed at the moment being predicted. And every model is compared against the honest null, not against a straw man: for player impact that is a league-average baseline, for game outcomes it is home-court advantage alone.

Most results use leave-one-season-out. The model trains on eight seasons and is graded on the ninth, so the coefficients that score a season never saw it. Where a within-season split is used instead, the model fits the earlier part of a season and predicts the later part.

Everything here rests on one thing being right. There is no public feed of five-man units for recent seasons, so we reconstruct them ourselves from play-by-play and box scores, stint by stint, for every game of nine seasons.

That reconstruction is checked against the league's own record: the per-player minutes we derive match the official box score to within half a second, in every game. If this step were wrong, every metric downstream would be wrong in a way no amount of clever modeling would reveal.

IPM is RAPM re-fit so each player is shrunk toward a box-score prior rather than toward zero. To test whether the prior actually helps, we held out stints and asked four models to predict them: IPM, pure RAPM, the box prior on its own, and a league-average baseline.

Nine seasons give nine within-season splits and eight consecutive season pairs. IPM posted the lowest error in all seventeen.

The honest half of that result: the gaps are small. A single stint is a handful of possessions, so most of the variance in the target is noise no model can reach. Read the ordering rather than the margin. The claim is not that IPM is dramatically more accurate, it is that it did not lose once to three alternatives that each had a plausible case for winning some of them.

The prior is graded separately, the same way. It predicts offense far better than defense, which is a limit of box scores rather than of our fit, and it is the single biggest weakness in the metric.

We publish two independent reads of team strength: Team IPM, built from the roster, and SRS, built from what actually happened on the floor. Ratings were fit on the first 60% of each season and graded on the final 20%.

The roster view predicts future margins better than the results view, which is the same finding Dunks & Threes reports for Team EPM, reproduced here on our own numbers. A fitted blend of the two scores better still, but least squares hands it a negative SRS weight because the two ratings are collinear. Predictive, but perverse as a published number, so it stays research rather than a column.

The pregame model is graded on 9,509 held-out games, leave-one-season-out across 2018-19 to 2025-26. Both ratings it uses are built only from games already completed at the moment of prediction.

The two ingredients trade places exactly where theory says they should. Early in a season, before the results rating has a sample, the roster prior carries the model. Later, the results rating takes over. The blend wins both windows.

Accuracy is the less interesting number. What matters for a probability is whether it means what it says: when the model claims 70%, does the home team win about 70% of the time?

Eight of the ten bins hold 150 games or more, and every one of those lands within 1.1 percentage points of its prediction. The 0-10% and 90-100% rows hold 7 and 56 games between them and are too thin to read anything into, which is why they are printed rather than hidden.

A validation page that only lists wins is marketing. These are the real limits, and they are on the pages that use these metrics too.

• Defense is the weak end of IPM. The box prior explains far less about defense than offense, so defensive IPM leans harder on the noisier on/off evidence.
• We have no player-tracking data. Screening, gravity and off-ball defense reach our numbers only through the on/off residual.
• The win model has no injury or rest information. Sportsbook closing lines, which have both plus market feedback, still grade better than we do. That gap is the honest cost of using only our own numbers.
• The results rating inside the win model advances by game rather than by calendar date, so a prediction can include other games played earlier the same evening. Worth about 0.2 points of accuracy.
• IPM is a season-level estimate. It does not decay within a season, so it will not track a midseason leap the way a daily-updating metric does.
• Single-season samples are small everywhere. When our metrics disagree with each other, the disagreement is usually the honest signal.