tools(GATE-GPU-LOCK-WRAPPER): one sanctioned way to take the GPU, and it records which lock it took (#587) - #596

… it records which lock it took (#587)

GPU serialisation on `dgx.casa` was split across two lock FILES. `$HOME/gpu.lock`
is what `.agents/benchmarking.md` and `.agents/environment.md` document and what
432 references in this tree take; `~/work/marlin442` took `/tmp/gpu.lock`. Two
jobs holding different files run concurrently while each believes it owns the
box, and nothing reveals it: `fuser $HOME/gpu.lock` shows an empty waiter list
while the other job is mid-run. On GB10 that is an OOM-reboot mechanism, because
unified memory makes a `gpu_memory_utilization` reservation HOST RAM, and it
voids any contention-sensitive number silently.

`scripts/gpu-lock.sh` is the one sanctioned way to take it, so the path stops
being something each script re-decides. `$HOME/gpu.lock` is canonical: it has
the callers and the documentation, and it is per-user, where a lock under `/tmp`
is world-writable -- any process on the box can hold or clobber the mutex
guarding someone else's measurement -- and is cleared by the very reboot it
exists to prevent, taking the evidence of who held it with it. `$GPU_LOCK` still
overrides, and the wrapper stamps whichever path it RESOLVED, so a divergence
shows up in the record instead of surfacing when two jobs collide.

Two properties are the design.

It REFUSES, never falls back. An untakeable lock aborts with exit 78 and the
command does not run; a bounded wait expires with its own exit 75 rather than
sitting forever behind a dead holder. A wrapper that proceeds on the assumption
it holds a lock it never took is one more instrument that cannot report its own
failure -- the class that produces a `Status: SUCCESS!` from a binary an ENOSPC
build never recompiled, and a "policy violation" that is really a checker unable
to write a temp file.

It RECORDS, and the exit code comes FIRST. Disk and load say the box was
unhealthy; the exit code says WHICH thing killed the run. `137` (SIGKILL --
another agent's `pkill`) versus a compiler diagnostic versus an ENOSPC line are
three different diagnoses and three different repairs, and only the first says
it was not your code at all. The stamp carries the resolved lock path, the
requested spelling, holder pid, wait, timeout, `df -h /` and load average at
acquire and again at exit, the command, and `exit-code` + `exit-reason` as the
first two fields of the release block.

The near-miss this is built from: a peer's same-binary A/B on this box survived
#587 only because its conclusion rested on `diff -r -q` over two output
directories -- byte-identity is contention-immune -- while the wall times
printed beside it were not. Had the conclusion rested on the times, #587 would
have voided it silently and nothing in the record would have said so. Stamping
does not prevent contention. It prevents contention being undetectable
afterwards.

Nesting passes through on the same resolved path instead of deadlocking on a
lock its own parent holds; a different path is still taken for real.

RED first: 48 tests written against an absent script, all 48 erroring, then
green. Eight mutations prove the checks catch their defect rather than
describing it -- the refusal returning instead of exiting, dropping the
`lock-path` field, dropping the `exit-code` field, swallowing the exit code,
collapsing the timeout status to 0, skipping the acquisition entirely,
unbounding the wait, and removing the pass-through so nesting self-deadlocks.
Each is caught.

THE SWEEP IS NOT CLOSED. The repo has 432 `$HOME/gpu.lock` references and zero
`/tmp/gpu.lock` -- but a THIRD spelling #587 did not name: `/tmp/gpu`, no
extension, taken as `exec 9>/tmp/gpu` by `scripts/dgx-online-serving.sh` and
`scripts/dgx-gdn-packed-component.sh`, named by `scripts/opt-dgx-gate.sh` and
`scripts/dgx-sglang-low-concurrency.sh`, and documented as this box's
`${GPU_LOCK}` in `.agents/coordination.md:99`. Those callers are outside this
change's authority and are owed a follow-up: until they are repointed, a run
under this wrapper does NOT exclude an online-serving or gdn-packed-component
run, which is #587 with different filenames. On dgx, `~/*.sh` converges on
`$HOME/gpu.lock` across 14 spellings that all resolve to one file, and
`~/work/marlin442`'s own scripts take NO lock at all; `/tmp/gpu.lock` did not
survive the 09:00 reboot, so what held it is not recoverable from the filesystem
and is not asserted here.

`.agents/benchmarking.md` and `.agents/environment.md` name the wrapper as the
sanctioned path, so the next agent does not read the old instruction and take a
raw `flock`.

Disk logged beside every result, per the box discipline: dev box 88% used /
56G free throughout; dgx 99% used / 61G free, read-only sweep only, nothing
built or deleted there.

Not wired into `scripts/agent-preflight.sh` SUITES or `.github/workflows/ci.yml`
-- both outside this change's authority, and owed.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]