mapping_reuse: release untracked VA when unmap stops at a foreign entry - #1354

Summary

reusemappingdbUnmap() can return without releasing untracked BAR1 VA, leaking address
space on every affected unmap. On a GPU with a small fixed BAR1 the aperture eventually
fragments to the point where dmaAllocMapping_GM107 cannot place a new mapping, which
surfaces as NV_ERR_NO_MEMORY at mapping_reuse.c:274, then Xid 31, then a display
engine hang.

The leak is invisible in nvidia-smi, whose BAR1 "Used" counts allocated memory rather
than reserved address space — which is why these failures appear at ~40% reported BAR1
usage and why closing GPU clients does not recover.

Where the untracked VA comes from

reusemappingdbMap(), in the !bNoReuse && bSingleRange path: when a cached entry
intersects the requested range without being an exact match, bAddToMap is cleared but
the function still falls through to pMapCb() and allocates new mappings. In the append
loop those entries take the else branch and are PORT_FREE()d without ever being
inserted into virtualMap or the physical map.

Those mappings exist in hardware and are tracked nowhere. The overhang unmap at the end of
reusemappingdbUnmap() is the only code that can reclaim them.

Why the overhang unmap is skipped

The walk had two different exits for the same condition:

if (!mrangeContains(range, revRange))
{
    if (bFirstRange)
    {
        break;      /* reaches the overhang unmap */
    }
    return;         /* skips it */
}

Both mean "this entry is not ours, stop walking". On the return path, everything in
[curOffset, mrangeLimit(range)) — untracked VA inside the caller's own range — is never
handed to pUnmapCb and stays reserved for the life of the address space.

The change

Replace bFirstRange with an unmapLimit that records how far the trailing unmap may
extend, and use one exit.

Beyond fixing the leak this addresses two latent issues on the path that already used
break:

• an entry straddling the end of the requested range could previously be unmapped while
  references to it remained; unmapLimit is clamped to that entry's start, so the unmap
  never reaches into it.
• the trailing size was computed as mrangeLimit(range) - curOffset behind a != guard;
  unmapLimit > curOffset cannot underflow if curOffset ever passes the limit.

Reproduction and measurement

RTX 2080 (TU104), BAR1 fixed at 256 MiB — Turing cannot resize it, so the standard
"enable Resizable BAR" workaround does not apply. Driver 610.57.04 open modules, Linux
7.2.3, KDE/Wayland.

Stock 610.57.04, one 1h53m session:

NVRM: dmaAllocMapping_GM107: can't alloc VA space for mapping.        (x89)
NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY]
      ... _reusemappingdbAddMappingCallback) @ mapping_reuse.c:274
NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY]
      ... reusemappingdbMap(&pBar1VaInfo->reuseDb, ...) @ kern_bus_gm107.c:3153
NVRM: Xid (PCI:0000:2b:00): 31, pid=1922, name=plasmashell, ... MMU Fault:
      ENGINE GRAPHICS GPC5 GPCCLIENT_GPCCS faulted @ 0x1_20010000 ... VIRT_READ
[drm:__nv_drm_gem_nvkms_map [nvidia_drm]] *ERROR* Failed to map NvKmsKapiMemory

The first failures appeared at a BAR1 high-water mark of 133 MiB (52%) and the session
ended in a display hang. Sampling every 5 minutes over the preceding six days recorded 23
such incidents, with VA failures accumulating from roughly 40% reported BAR1 usage.

With the unmap fix applied, same workload and same desktop session:

uptime                   1h03m
BAR1 high-water mark     183 MiB (71%)
can't alloc VA           0
Xid                      0
Failed to map NvKms      0
mapping_reuse asserts    0

The patched module sustained 50 MiB beyond the level at which the stock module was already
failing, with no VA allocation failures.

One disclosure: the measured build carried only the minimal return → break change. In
the leaking case this PR's form is identical to what was tested; the unmapLimit clamp
only narrows the trailing unmap, and is there to keep the straddling-entry case safe.

Related

• RTX 5070 (GB205): __nv_drm_gem_nvkms_map composes a mapping that spans BAR1→BAR3, causing mapping_reuse.c:273 NV_ERR_NO_MEMORY and krcWatchdog GPU lock — driver 595.71.05 (open kernel modules), Resizable BAR disabled #1132 — GB205, Resizable BAR disabled, same NV_ERR_NO_MEMORY at mapping_reuse.c
  followed by a GPU lock.
• mapping_reuse: correct range size when unmapping failed allocations #948 fixed the unmap size in the err_unmap cleanup path of reusemappingdbMap(); this
  is a separate path in the same file.

The bAddToMap behaviour described above is left alone here — reusing partially
overlapping mappings likely needs a design decision rather than a local fix, and this
change makes the VA it produces reclaimable. Happy to follow up on it separately.