Scoring a model you do not ship

Austin Wagner

Three ordinary training decisions compose into publishing a test score for a model that never leaves the machine. It was in my own pipeline: the manifest carried a validation loss from epoch 17 beside a test loss measured on epoch 20, neither labelled. I measured it, fixed it, and confirmed the fix on the cluster.

Problem
Best-checkpoint selection and final-model evaluation look independent. They are not, and nothing errors when they disagree.
Result
The reported score now describes the bytes that get uploaded, verified on a run where the best epoch was not the last.
Cost of the defect
0.0011 test loss, in the direction the mechanism predicts, on the one arm measured both ways.
Stack
PyTorch, GPT-2, Slurm on a university cluster, digest-pinned container images.

Three correct decisions, one wrong model

Each of these is defensible alone. The best-checkpoint save overwrites the output directory on every validation improvement, so after the last improvement that directory holds the best epoch. The final-model save is skipped when a best checkpoint exists, which is right, because it would overwrite the selected model with a worse one. Test evaluation then runs against the model still resident in memory.

That resident model is the last epoch, not the selected one. So a run with a holdout ends with one model on disk and a different one in memory, and reports a number for the one nobody receives.

Nothing fails. Early stopping behaved correctly, the checkpoint machinery worked, the evaluation worked. There is no error state anywhere, which is why the only thing wrong is which object the second feature was pointed at.

The test was harder to write than the fix

The fix is one idea: reload the selected weights before scoring. Reloading from disk rather than from an in-memory copy is the stronger form, because it also proves the artifact is loadable, so a truncated save surfaces at scoring time instead of in whatever consumes it months later.

Testing it needs a fixture whose best epoch is not its last, and that is the difficult part. A uniform corpus drives validation down monotonically, measured at 0.3326, 0.0231, 0.0046 and 0.0024 across four epochs. When the best epoch is the last one, disk and memory agree whether or not the restore runs, so the test passes against the very defect it exists to catch.

What worked was twelve training lines over thirty epochs: small enough to memorise, long enough for validation to turn. The test now asserts that premise explicitly and reports itself blind if a later change flattens the curve:

    best_epoch = min(epochs, key=_val_at)
    if best_epoch == epochs[-1]:
        raise AssertionError(
            f"fixture's best epoch is its last ({best_epoch}); "
            f"val curve {(...)} "
            "cannot distinguish a restored model from an unrestored one"
        )

services/Model-Trainer/tests/test_base_trainer_best_checkpoint.py, the premise guard, one expression elided.

What it cost, and what that does not prove

The same arm, same configuration, before and after: reported test loss 0.8634 against 0.8623. The published number had been 0.0011 too high, in the direction the mechanism predicts, since a non-selected epoch scores worse than the selected one.

That delta is not attributable to the fix. The two runs were in different environments, one under a conda environment and one inside the pinned image, and I made no attempt to separate the two effects. What is attributable is which model got scored, because the log names it: the restore is recorded three seconds before the test result.

Both branches were exercised in one submission rather than asserted separately in unit tests. One arm had a best epoch earlier than its last, so the restore had to fire. The other runs with no holdout, so there is no checkpoint and the restore must correctly do nothing.

Record

Every figure above is bound to a named section of the wiki page shipped-checkpoint-and-reported-score-can-be-different-models and checked mechanically before this page can deploy. That page carries the scheduler job IDs, the artifact digest and byte count, and a section naming which evidence supports which claim.

Known limits. One run on each side of the fix, so the agreement of the two best-validation losses to four decimals across a conda environment and a container is an observation rather than a result, and the page says so. The comparability consequence, that two arms shipping different epochs are not the same kind of object, is a reading rather than something either manifest records. Every figure and every mechanism claim on this page was re-checked against the current training code on 2026-09-14 and holds.

Sources