What Win Rate Misses in Sports Prediction Models
Two hypothetical models pick the same seven winners from ten games. A worked Brier-score example shows why their probabilities still differ in quality.
Two prediction models choose the same team in ten games. That team wins seven times. Both models can report a 70% win rate.
One model gave the team a 70% chance in every game. The other gave it a 95% chance. The second model expressed much greater confidence, but the win-rate headline cannot tell you that.
This hypothetical example shows why evaluating a sports prediction requires the probabilities behind the picks.
The same picks, different claims
For simplicity, suppose every game has two possible outcomes: the selected team wins or loses. This is a binary example; a sport or market with a draw would need a separate draw probability.
Here is the complete setup. These are invented forecasts and results for illustration.
| Across ten games | Model A | Model B |
|---|---|---|
| Probability assigned to the selected team | 70% every time | 95% every time |
| Times the selected team wins | 7 | 7 |
| Times the selected team loses | 3 | 3 |
| Win rate | 70% | 70% |
Model B has not picked more winners. It has made a stronger claim about how often its selected teams should win. To assess that claim, we need a score that retains the forecast probability.
Work through the probability errors
For a binary event, the Brier score averages the squared difference between the forecast probability and the outcome. Record a win as one and a loss as zero. Under this binary convention, a perfect score is zero and lower is better. The scikit-learn Brier-score documentation sets out the definition.
For Model A:
- Each win contributes (0.70 − 1)² = 0.09.
- Each loss contributes (0.70 − 0)² = 0.49.
- Across seven wins and three losses, the average is (7 × 0.09 + 3 × 0.49) ÷ 10 = 0.2100.
For Model B:
- Each win contributes (0.95 − 1)² = 0.0025.
- Each loss contributes (0.95 − 0)² = 0.9025.
- Its average is (7 × 0.0025 + 3 × 0.9025) ÷ 10 = 0.2725.
| Forecast | Win rate | Binary Brier score |
|---|---|---|
| Model A: 70% probabilities | 70% | 0.2100 |
| Model B: 95% probabilities | 70% | 0.2725 |
| Reference: 50% for every game | — | 0.2500 |
Model A scores better on this sample. Model B even scores worse than the reference forecast that assigns 50% to every game. Its three confident misses outweigh its smaller errors on the seven wins.
The reference row has no win rate here because a 50–50 probability alone does not select a winner.
What this example establishes
The calculation establishes a difference in probability error on these ten games. It does not establish that Model A will be better on future games, or that its probabilities are calibrated generally. Ten outcomes are too little evidence for that conclusion.
Calibration asks whether events assigned similar probabilities occur at roughly those frequencies over a sufficiently informative sample. A single correct prediction cannot answer that question. The calibration guide also explains why an overall Brier score reflects more than calibration alone.
A fuller evaluation needs enough forecasts, a relevant baseline, and results across different periods. Those forecasts must have been produced using information available before the events. Otherwise even a carefully calculated score can describe an unrealistic test.
What to look for behind a performance headline
When reading a claim about a sports model, ask for four things: the exact event being predicted, the probabilities issued before it, the evaluation period and sample size, and the comparison baseline. A win rate without that context leaves the model's confidence unexamined.
This matters for the prediction technology Cognora will develop for PicksHub. Our partnership announcement describes the collaboration; the worked example above is an explanation of evaluation, rather than a result from that project.
Next, follow a hypothetical match-day timeline to see how future information can slip into a historical test.