# Kneeslide forecast backtest — November 5, 2026

## Decision

Version 2 uses K=20, a 0-point adjustment in Elo rating updates, and 10% offseason regression toward 1,500. These were selected by the lowest training log loss, before evaluating the audit seasons. The best setting in this finite grid is not a precisely established optimum. The evidence does **not** establish a meaningful accuracy improvement over v1. Both settings should remain versioned, inspectable model choices.

A zero Elo home adjustment does not remove home advantage from match forecasts. Separate home and away goal rates continue to account for venue. The grid only tests the additional adjustment used when updating ratings after results.

## Design

- Warmup: 2014–15 through 2018–19 (1,900 matches).
- Selection: 2019–20 through 2022–23 (1,520 matches). Every forecast uses pre-match features; results update ratings only afterward.
- Retrospective audit: 2023–24 through 2025–26 (1,140 matches). These seasons were excluded from final selection, but were already inspected under v1. This is not a new blind test.
- 4,800 combinations: K in 10/20/30/40; Elo home adjustment in 0/30/60/90/120; offseason regression in 0/10/20/30/40/50%; crossed with the existing 40 scoring-model combinations.
- Six expanding-window checks: select using preceding seasons only, then predict the following season. A comparator fixes the original Elo settings but retunes scoring weights on the same preceding seasons. These checks are diagnostic and do not alter the final selection.
- Season simulations at 0, 90, 190 and 280 completed fixtures; 2,000 simulations per cutoff. Five strength-uncertainty settings (0/35/70/105/140 Elo) tested on training seasons only. Select by mean 80% interval score, penalizing both misses and excessive width.
- 240,000 season-backtest simulations across candidate and original models; production reconstructions use 10,000 each. Sporting points before deductions are the season-backtest target.
- Input SHA-256 hashes, all sensitivity profiles, top 20 training candidates, and paired bootstrap intervals are in the JSON download.

## Match results

| Metric, 2023–26 audit | Original | Updated |
|---|---:|---:|
| Outcome log loss (lower better) | 0.984063 | 0.983871 |
| Multiclass Brier (lower better) | 0.586766 | 0.586835 |

The log-loss gain is about 0.02%; Brier is slightly worse. The updated model improves audit log loss in one of the three seasons. This is mixed evidence, not a reliable uplift.

| Season | Original log loss | Updated log loss |
|---|---:|---:|
| 2019-20 | 0.974498 | 0.973076 |
| 2020-21 | 1.024554 | 1.024901 |
| 2021-22 | 0.955093 | 0.952295 |
| 2022-23 | 0.976154 | 0.978361 |
| 2023-24 | 0.933327 | 0.930692 |
| 2024-25 | 0.987867 | 0.988189 |
| 2025-26 | 1.030994 | 1.032732 |

The paired match-bootstrap 95% interval for updated-minus-original log loss is -0.001683 to +0.001324. A season-block bootstrap gives -0.002636 to +0.001738, but only three audit seasons are available. Both include zero. These are descriptive uncertainty estimates, not proof of equivalence.

## Parameter sensitivity

Each row retunes the other parameters using training results. Audit numbers below are reported afterward and were not used to choose the final values.

| Offseason regression | Training log loss | Audit log loss |
|---|---:|---:|
| 0% | 0.982564 | 0.984040 |
| 10% | 0.982158 | 0.983871 |
| 20% | 0.982556 | 0.984091 |
| 30% | 0.983130 | 0.985250 |
| 40% | 0.983978 | 0.986356 |
| 50% | 0.984239 | 0.985920 |

| Elo home adjustment | Training log loss | Audit log loss |
|---|---:|---:|
| 0 | 0.982158 | 0.983871 |
| 30 | 0.982202 | 0.983888 |
| 60 | 0.982296 | 0.983995 |
| 90 | 0.982479 | 0.984233 |
| 120 | 0.982583 | 0.984204 |

The home-adjustment profile is especially flat: the data do not strongly identify an exact value. More aggressive offseason regression generally performs worse within this grid. K remains 20; scoring weights remain beta=0.6, weight=0.5, xG mix=0.5 and low-score correction=0.

## Expanding-window checks

| Next season | Retune all settings | Fix old Elo, retune weights |
|---|---:|---:|
| 2020-21 | 1.033688 | 1.032835 |
| 2021-22 | 0.949105 | 0.955093 |
| 2022-23 | 0.986219 | 0.976154 |
| 2023-24 | 0.930692 | 0.933327 |
| 2024-25 | 0.987558 | 0.987867 |
| 2025-26 | 1.032122 | 1.027945 |

Mean log loss: 0.986564 for retuning all settings, versus 0.985537 with fixed old Elo. Selected settings vary considerably between windows; frequent retuning is not demonstrably beneficial.

## Season projections

Training interval scores select 70 Elo points of persistent strength uncertainty, unchanged from v1. The updated training coverage is 81.6%.

| Audit metric, all four cutoffs | Original | Updated |
|---|---:|---:|
| Mean absolute final-points error | 6.867 | 6.856 |
| 80% interval coverage | 78.3% | 77.5% |
| 80% interval score, lower better | 27.746 | 27.912 |

| Completed fixtures | Original points MAE | Updated points MAE | Updated 80% coverage |
|---|---:|---:|---:|
| 0 | 10.452 | 10.398 | 73.3% |
| 90 | 7.877 | 7.870 | 75.0% |
| 190 | 5.338 | 5.355 | 78.3% |
| 280 | 3.804 | 3.801 | 83.3% |

Cutoffs from the same season are correlated, so these are not 12 independent season tests. The preseason ranges still under-cover. With only 2,000 runs per cutoff, small changes can include simulation noise.

## Effect on the current forecast

Using identical live inputs, 10,000 runs and seed 538, the mean absolute projected-points change is 0.61; the largest is 1.47. Arsenal changes from 84.55 to 85.83 points and Manchester City from 77.26 to 78.73. Title probability changes also contain Monte Carlo noise; see the impact JSON.

## What remains uncertain

- This searches a finite grid, not every possible model. Goal-rate priors, 0.96 match decay, offseason decay of goal/xG history and the 1,500 initialization of unseen clubs are retained assumptions.
- Historical xG comes from different providers and has incomplete coverage. Historical data may contain corrections unavailable at the original prediction time.
- Older returning clubs retain their previous state; this is not a separate promoted-team model.
- The season simulator uses discrete, capped Elo-shift lookup tables. Injuries, transfers and tactical changes are not individually modeled.
- Model selection uses a small number of seasons. The current season was excluded from tuning, but no new prospective performance claim can be made yet.

## Deployment and reproducibility

The predictor and match simulator use model 2.0.0. Existing v1 snapshots remain unchanged at their original URLs; their manifest and model configuration are retained. New v2 matchday snapshots live in a separate versioned directory and are labeled reconstructed. Future snapshots are immutable within that version.

Run from the repository root with Python + NumPy and Node 22+:

```sh
python scripts/backtest-elo.py
FORECAST_BASELINE=1 node --experimental-strip-types scripts/build-forecast.mjs
node --experimental-strip-types scripts/build-forecast.mjs
node --experimental-strip-types scripts/upgrade-forecast-model.mjs
node --experimental-strip-types scripts/forecast-model-impact.mjs
python tests/forecast-calibration.test.py
node --experimental-strip-types --test tests/forecast.test.mjs tests/forecast-snapshots.test.mjs
```

## Data and detailed results

- [Match audit, sensitivity and input hashes](backtest-2026-11.json)
- [Original season backtests](backtest-season-original.json)
- [Updated season backtests](backtest-season-updated.json)
- [Current forecast impact](backtest-impact.json)
- [Original snapshot manifest](snapshots/index-v1.0.0.json)
- [OpenFootball results](https://github.com/openfootball/england)
- [Historical xG source archive](https://github.com/vaastav/Fantasy-Premier-League)