How many compartments fit in one model

A serving design assumed several separately trained compartments could be wired into one request. Measuring it first put the naive limit at two, then a training recipe moved four-compartment retention from −45.4% to +44.6%, and a diverse pool of companions has since taken it to +55.5%.

Question: Can one model hold several specialisations that stay distinct and stay separately addressable?
Result: Under naive training, no: past two compartments the primary's gain is erased. Train the base to expect a crowd and the limit moves; at 48 layers the eight-compartment penalty vanishes.
Method: A sweep with its own artifact detectors, replicated across three roster rotations, with a bit-identity certificate: two separate processes produced SHA-256-identical records.
Stack: Python, PyTorch, pretrained GPT-2 at three depths, an RTX 3090 Ti offline and V100s on the cluster.

The controls caught the result before a reader did

This is the part I would want to be asked about. The first roster read +27% retention at eight compartments: a good number, in the direction the design wanted, and wrong.

It was leakage. Compartments trained on two technical corpora were predicting a third corpus they had never seen, because all three share vocabulary (+0.18 and +0.41 cross-gain). The sweep scores every foreign compartment alone against the primary's held-out text precisely to catch that, so the artifact surfaced as a measurement rather than as a conclusion. The clean rerun put eight compartments at −7.0%.

CROSS-GAIN ARMS GUARD THE PREMISE. Every retention here is only meaningful
if the other corpora are genuinely unrelated to the primary -- the two-halves
artifact (...) is the standing warning. Each
other-corpus cartridge is therefore scored alone on the primary held-out
items, and a positive cross gain marks the retention it participated in as
inflated. The check rides in the record, not in a caveat somebody remembers.

services/Model-Trainer/src/model_trainer/cli/cartridge_composition_sweep.py, module docstring, one parenthetical elided.

Moving the limit rather than reporting it

A ceiling is only a finding if you try to break it. The published literature names the escape route (concatenation works when it is trained for, not when it is assumed), so the next run trained the base to expect company instead of training each compartment alone. Retention of the primary compartment's held-out gain, four and eight compartments wired at once:

Recipe	Model	Four	Eight
Naive, each compartment trained alone	GPT-2 small	−45.4%	−7.0%
One trained companion present at p=0.5	GPT-2 small	+44.6%	+26.5%
Three companions on three held-out corpora	GPT-2 small	+55.5%	+28.0%
Same pool, language-model objective	GPT-2 XL, 48 layers	+54.8%	+54.8%

The naive numbers stand as the baseline every recipe is judged against. At eight compartments on the small model the composed arm equals its untrained control, so what remains is slot dilution rather than content interference; at 48 layers eight equals four to a tenth of a point, so the penalty is a mid-depth valley, not a law.

What this does not settle

The recipe does not survive the jump to a 7B architecture as-is, and the reason is the measurement's precondition rather than composition: a single compartment's gain there is a mean of +0.16 across nine seeds with one seed negative, against +0.78 to +0.86 on every GPT-2 rung, so a retention ratio at 7B divides by a number that barely exists. The plain 7B base already predicts each corpus at the level the smaller bases reach only with a compartment.

Every compartment measured here is a corpus. Whether the same curve holds when the compartments are dispositions (personas rather than bodies of knowledge) is open, and the published work suggests it is harder. That is the next experiment, deliberately unstarted rather than quietly skipped.

Record

Every figure above is bound to a named section of model-trainer-composition-ceiling or model-trainer-companioned-training-recipe and checked mechanically before this page can deploy. Both are pinned to the git blob hashes of the source files that produced the numbers, so a change to the measurement code invalidates the citation rather than silently outdating it, and both are public.

Known limits. These are retention ratios on held-out text, not a serving benchmark, and nothing here times a request. Until 2026-09-14 this page said "one model family at one scale, on a single card"; the programme had by then measured three depths of GPT-2 and a 7B architecture on cluster V100s, and the page now says so. The wider programme has had published claims withdrawn for sitting below what its question set could resolve; the figures here are ratios with margins in the tens of points, and the withdrawal is recorded in the wiki.

Sources

The naive ceiling: wiki/pages/model-trainer-composition-ceiling.md; the recipes, the diverse pool, the scale ladder and the 7B result: model-trainer-companioned-training-recipe.md.
The sweep whose docstring is quoted above and the companion sweep: cartridge_composition_sweep.py, cartridge_companion_sweep.py, cartridge_diverse_companion_sweep.py.
Every run summary, in order, under the mi project: docs/RESEARCH.md.
The literature survey of what composition permits is a private wiki page, many-personas-in-one-model-feasibility.
This article's manifest: provenance.json.