Reproducible GPU inference

Austin Wagner

One model and one set of weights produced byte-identical output on 8 NVIDIA GPU models across 6 architecture families, two operating systems and two CUDA toolchains.

Problem
PyTorch's determinism settings make one card reproduce itself. Nothing in them makes two different cards agree with each other.
Result
2,627 of 2,627 per-item scores bit-identical on every pair of cards. One outcomes file, one SHA-256, six cards, two operating systems.
Cost
Free within measurement noise at any batch from 128 rows up.
Stack
PyTorch, CUDA, cuBLASLt, GPT-2; V100, A30, A100, L40S, RTX 3090 Ti, 3070 Ti, GTX 1630, RTX PRO 6000.

What was actually wrong

The disagreement looked like a threshold: cards agreed below a certain model size and diverged above it. That reading was wrong, and finding out cost the most time.

Running matrix multiplications in isolation and fingerprinting the whole output tensor rather than the final loss showed six of eight shapes producing three different tensors on three cards, including shapes whose reported losses were bit-identical. The cards were never computing alike. The difference simply did not survive being reduced to one number.

So the threshold was not where cards begin to disagree. It was where a disagreement already present grew big enough to move the last bit of a number someone reads.

Three interventions, and what each was worth

InterventionWhat it fixedLeft over
Remove the vendor's split-K reductionisolated matmuls 2 of 8 → 8 of 8attention untouched
Pin attention to its math backend198 of 900 score comparisons exactthe projections still differ
Own the reduction order2,627 of 2,627, every pairone Turing card, since closed

The first two are configuration. The third is not: it stops asking the vendor library for the sum and accumulates in a fixed order, so the arithmetic is the program's rather than the device's. The kernel package states the contract:

THE ORDER CONTRACT, WHICH IS THE WHOLE PACKAGE. Every output element is
produced by ONE thread as a strictly ascending-k sequence of
``acc = acc + x*w`` -- separate multiply, separate add, two roundings --
(...) Compiled with ``--fmad=false`` so the
compiler cannot contract the pair into one fused rounding, and with no
fast-math anywhere so it cannot reassociate. Parallelism lives ACROSS
output elements; within one element nothing is parallel, which is why SM
count, tile scheduling and occupancy -- everything that differs between
cards -- cannot touch the bits.

clients/OrderedKernels/src/ordered_kernels/kernels.py, module docstring, one clause elided.

A GTX 1630 then broke the identity at 147 of 150, the only card ever to. The three divergent items were not the long ones; one was the shortest in the set. A stage probe at those exact sequence lengths named the operation: the value product inside attention, at 15- and 16-token sequences. Owning that closed the card.

What it costs

Between 0% and 13% per matrix multiply at a realistic batch size, several measurements negative. An earlier version of this measurement reported up to +84% and argued for the wrong decision: it was taken at a single 64-token sequence. Sweeping the batch dimension shows the cost collapses between 64 and 128 rows and never returns: one doubling, and any real workload is already past it.

What the result does not establish

A run with no divergences bounds a rate; it does not establish zero. The exact Clopper-Pearson bound on the tightest row is 0.114%, and pooled across the battery 0.072%. Measured against the scale this operates at, where one divergence in a full 2,627-item run is 0.0381%, none of those rows excludes it.

Bit-identity across a corpus also says nothing about whether a fresh pair of runs reproduces it. That was observed once.

Record

Every figure above is bound to a named section of a wiki page and checked mechanically before this page can deploy. The wiki carries the per-figure footnotes, the container image digests, the scheduler job IDs and the controls that were run rather than assumed: a-loss-agrees-where-the-computation-does-not and cross-gpu-agreement-is-recoverable-by-disabling-split-k.

Known limits. Timing figures are per-GEMM, not end-to-end; nothing here times a full forward pass or a training step. The isolated-matmul campaign records its container as content-addressed but does not carry that image's digest, so those runs are checkable by date, driver and card rather than by digest. Until 2026-09-14 this page said nine GPUs across five generations; the source page's own roster names eight models across six families, and it had miscounted the cards while this page had miscounted the families.

Sources