Submitting to a shared GPU cluster, safely

Austin Wagner

8 research projects run on a shared university GPU cluster through one tool that will not let a job be built if it would fail, and can answer afterwards which job produced which artifact.

Problem
UCI's HPC3 is a Slurm cluster with 102 distinct users holding jobs when this was measured against it. Its expensive failure modes are invisible to the obvious check.
Result
Failure modes encoded as rules that execute rather than documentation somebody has to remember, so a job that would break one cannot be constructed.
Scale
17,432 lines of source against 19,849 lines of tests; 15 commands over one workspace document.
Stack
Python, Slurm, containerised environments pinned by digest.

Five conditions that look like health

Each of these is invisible to the check anyone would actually run, which is why a job can burn a week without anything reporting a problem. Three existed at first; the fourth and fifth were built the same day, and the fourth found something on its first run.

ConditionWhat it looks likeWhy the obvious check misses it
blockedPending, for a reason that will never resolve261 of 621 pending GPU jobs sat on DependencyNeverSatisfied, which in squeue's state column is indistinguishable from healthily waiting on resources
unaccountedSubmitted here, unknown to the clusterNo cluster-side query can find these: the evidence is the absence of a cluster-side record, which is the whole reason a local ledger exists
unclaimedRunning under the account, unknown to the ledgerThe mirror of unaccounted, and the direction that went unbuilt: 21 image builds had run unrecorded through a documented raw sbatch, and the check found the twenty-second on its first run
silentRUNNING, holding GPUs, log no longer growingLog age is measured against the cluster's own clock, because a few minutes of skew would either invent staleness or hide it
oversizedAsks for far more wall clock than its work has ever takenSlurm backfills a job into a hole its own size, so it waits for one it never needed. Caught a project declaring 720 minutes for members that finished in 27
 
CLOCK_PROBE = r'echo "now $(date +%s)"'
"""Reads the cluster's own clock. Every batch carries one -- see below."""

tools/hpc3/src/hpc3/core/logs.py, the clock probe the silent check reads instead of this machine's time.

Rules that run

Preflight is non-skippable. Submission probes the environment, uploads the real rendered script, and asks the scheduler to test it by path. There is no flag to skip it and no code path that reaches the cluster without it, and the same rendered file is then submitted, so preflight and submission cannot drift.

A budget, because nothing else says stop. The cluster's quality-of-service limits bound what runs at once; nothing bounds the total, and the free partitions do not bill. A 24-GPU three-day sweep is 1,728 GPU-hours, inside every limit the cluster enforces, and not a reasonable share of a shared machine. So the tool enforces one, before submission and again while running.

Both rules exist because the cluster will happily let you do the wrong thing. Neither is a warning the user can dismiss.

Which job produced this file

Four claims are enforced rather than asserted: the environment is the pinned one, the staged bytes are the published ones, the run's numerical determinism is declared and recorded, and a result traces back to the run that made it. The ledger is an index from job to image to artifact, so a hash off a results file answers which job trained this?

That last one is what makes a negative result trustworthy. A null that cannot be traced to a specific image and a specific job is not a finding, it is a file.

Record

Every figure above is bound to a named section of a wiki page and checked mechanically before this page can deploy: the counts to hpc3-cli, the cluster measurements to the package's own wiki, which is public and linked below.

Known limits. The 102-user and 261-of-621 figures are single observations of a shared cluster at one moment, not averages; they say the condition is common enough to design for, not how common it is. The line counts were re-measured on 2026-09-14 and decay: each moved by several hundred lines in the three days before that. This page said "three conditions" until 2026-09-14; the fourth and fifth had existed for seventeen days.

Sources