Testing Sports Prediction Models Against the Clock

Suppose a model forecasts a match at 2 p.m. The starting lineup is announced at 4 p.m. Kickoff is at 6 p.m.

A month later, an analyst downloads a dataset containing that lineup and tests the model on the match. The record looks historical: the fixture is over, and every field belongs to the right game. But the lineup was unavailable when the 2 p.m. prediction was supposed to happen.

That is a concrete route to data leakage. A historical test can reward information that the real forecast could never have used.

Reconstruct one match day

Consider this hypothetical sequence. The times are illustrative, all on the same day and in the same local time zone.

The dividing line is availability at prediction time. Attaching a later record to the correct fixture does not move that record back in time.

The scikit-learn leakage guide describes the general problem: a test becomes unrealistically optimistic when the model gets information it would not have during real use.

Store when information became usable

A useful input record needs more than a match identifier. At minimum, distinguish:

• Event time: when the underlying event happened.
• Publication time: when the provider released the information.
• Receipt time: when the forecasting system received it.
• Revision identity: which version of the record the system saw.

Suppose the provider published the injury update at 1:15 p.m., but the forecasting system did not receive it until 2:05 p.m. A replay of that system's 2 p.m. forecast must exclude it. Publication time alone would make the replay more informed than the original system.

Likewise, overwriting yesterday's record with today's correction destroys evidence of what was available. Preserve the earlier version, or retain enough history to reconstruct it.

Build the training window around available outcomes

The training side has a cutoff too. A match that started before 2 p.m. but ended afterward cannot supply its final result to a model trained at 2 p.m.

A practical historical replay can use this sequence:

1. Choose a forecast cutoff.
2. Select training examples whose outcomes were known before that cutoff.
3. Reconstruct eligible inputs from the versions available at the cutoff.
4. Fit the model and its preprocessing using only the training data.
5. Record the forecast, then score it after the outcome becomes available.
6. Advance the cutoff and repeat.

Chronological evaluation is the principle behind tools such as TimeSeriesSplit. Its documentation also specifies assumptions about equally spaced samples. Sports fixtures are irregular: five games could cover one afternoon or several weeks. Calendar-based evaluation windows and explicit availability rules may be more appropriate than applying a row-based splitter unchanged.

Check the less obvious leaks

Removing the final score from the input columns is only a first check. A season average calculated after the season can contain the match being predicted. A feature-selection step run on the entire dataset can incorporate test outcomes. A missing-value replacement fitted across training and test periods can use information from the future.

Fit learned preprocessing inside each training window, as the common-pitfalls guide recommends. Keep the final evaluation period separate from repeated tuning decisions.

For each forecast, preserve the model version, cutoff, input versions, output probabilities, and eventual result. That record gives someone else a way to inspect the test rather than taking a performance summary on trust.

Ask what the score actually measures

An earlier forecast and a later forecast can both be legitimate products. They answer different questions if one has access to a confirmed lineup and the other does not. Compare models at the same cutoff, against a baseline with the same information access.

The Cognora–PicksHub partnership concerns technology for PicksHub's prediction models. The timeline here illustrates an evaluation problem that work will need to address; it is not a description of a deployed system.

Once that timing boundary is sound, the next question is how to score the probabilities themselves.