Measuring what a corpus installs

Establishing that an unexposed model already answers 52.3% of the items, against 25.0% by chance, is what makes every later number interpretable. Surrounding prose constrains a masked value far more than the chance baseline suggests, so an intervention that lifts raw accuracy to sixty percent has done almost nothing.

A floor established after the results is a floor chosen with knowledge of the answer.

The marker intervention is inert here under all three conditions its source specifies, supplied individually and together. That is a published result failing to replicate on a different corpus type, and reporting it plainly is the point.

An earlier version of this reported markers as a real effect, on a one-sample t of 14.0 over three per-seed differences. That was an artifact. The arms share one deterministic item set, so the same items are consistently easy or hard, and the small spread was measuring that correlation rather than the effect.

The paired test over all 2,627 items finds nothing. The finding was retracted and the page rewritten. This is precisely the failure mode paired testing exists to prevent, and I walked into it before catching it.

Writing the controls section surfaced a real defect in the training service: the corpus splitter divides by file, and its single-file branch returns the same file as train, validation and test. Every arm's corpus is a single file, so every validation and test loss in every run manifest is a training-set loss, and so is every end-to-end training test in that service, 91 fixtures deep.

The published results are unaffected, and that is checkable rather than asserted: no arm uses validation loss for anything, and the cloze set is the only cross-arm metric. The defect was pinned with a test that declared it known, so a fix would turn the test red; the fix landed three days later, splitting a one-file corpus by line, and that test became the fix's own:

def test_split_corpus_single_file_partitions_its_lines_disjointly(tmp_path: Path) -> None:
    """The headline fix: one file yields a real holdout, not itself three times.

    (...) Asserted as the exact three tuples, because the defect
    this replaces -- returning the whole corpus as all three partitions -- also
    satisfies "validation is non-empty".
    """

services/Model-Trainer/tests/test_corpus_and_dataset_edges.py, one sentence elided; the fix is commit 3ce75c2f0.