Skip to content

uvm: Guard bottom-half queue flush on suspend - #1340

Open
tarik0 wants to merge 1 commit into
NVIDIA:mainfrom
tarik0:uvm-suspend-bottom-half-guard
Open

uvm: Guard bottom-half queue flush on suspend#1340
tarik0 wants to merge 1 commit into
NVIDIA:mainfrom
tarik0:uvm-suspend-bottom-half-guard

Conversation

@tarik0

@tarik0 tarik0 commented Sep 5, 2026

Copy link
Copy Markdown

Fixes #1212

Problem

On GPUs without replayable-fault support (Maxwell), suspending while any process holds /dev/nvidia-uvm open crashes the kernel and hangs the machine:

BUG: kernel NULL pointer dereference, address: 0000000000000000
RIP: 0010:_raw_q_flush+0x87/0x110 [nvidia_uvm]
 nv_kthread_q_flush+0x18/0x70 [nvidia_uvm]
 uvm_suspend+0x17b/0x1a0 [nvidia_uvm]
 uvm_suspend_entry+0xb6/0xf0 [nvidia_uvm]
 nv_uvm_suspend+0x32/0x50 [nvidia]
 nv_set_system_power_state+0x329/0x510 [nvidia]
 nv_procfs_write_suspend+0x129/0x160 [nvidia]
note: nvidia-sleep.sh exited with irqs disabled
note: nvidia-sleep.sh exited with preempt_count 1

The crashed task still holds the queue spinlock, so the suspend that follows fails to freeze user space and the system stays hung with the display already off. Reproduces 100% with a retained GPU; suspend only succeeds while no UVM client exists, which is why it looks intermittent.

Cause

uvm_parent_gpu_init_isr() creates isr.bottom_half_q only inside if (parent_gpu->replayable_faults_supported), and sets isr.replayable_faults.handling = true on the same path. uvm_maxwell.c sets replayable_faults_supported = false, so the queue stays zero-initialised. uvm_suspend() flushes it for every retained GPU regardless, and _raw_q_schedule() does list_add_tail() on a NULL list head.

Fix

Guard the flush with isr.replayable_faults.handling, the flag set by the code that creates the queue. uvm_suspend() already guards the kill_channel_q flush on the next lines with non_replayable_faults.handling; this makes the two consistent. Every GPU supported by the open kernel modules sets the flag, so their path is unchanged. The defect matters for the shared UVM source in the R580 proprietary driver, the last branch supporting Maxwell, Pascal and Volta. Please carry the fix into R580.

Testing

GPU GeForce GTX 970 (GM204, Maxwell)
Driver 580.173.02, DKMS build (nvidia-580xx-dkms, Arch/CachyOS)
Kernel 7.1.3-1-cachyos
Sleep mem_sleep = deep (S3), NVreg_PreserveVideoMemoryAllocations=1, nvidia-suspend.service

Without the patch, systemctl suspend with a UVM client open produced the trace above and needed a hard reset (last occurrence in the same session, module build ID confirmed unpatched).

With the patch applied to the DKMS source and the rebuilt module verified loaded by build ID: 4 suspend/resume cycles in one boot, all clean. Two with VS Code holding /dev/nvidia-uvm; two with a CUDA driver-API program doing continuous work on a 16 MiB cuMemAllocManaged buffer (PTX kernel increments every element once per second, host verifies each iteration). The program ran through both suspends without interruption: 195 iterations, 0 mismatches, values consistent across the sleep gaps. Journal per cycle:

nvidia-suspend.service: Deactivated successfully.
PM: suspend entry (deep)
Freezing user space processes completed (elapsed 0.001 seconds)
PM: suspend exit
nvidia-resume.service: Deactivated successfully.

No nvidia_uvm errors, warnings or oopses in the kernel log for the whole boot. Tested on Maxwell only; no Turing-or-newer GPU available for a control run, but those GPUs already take the guarded path with handling == true.

Also reported on the forums: GM108M, 580.159.03, kernel 7.0, 545.29.02, kernel 6.6.

uvm_parent_gpu_init_isr() only creates isr.bottom_half_q when the GPU
supports replayable faults. uvm_suspend() flushes that queue for every
retained GPU without checking, so on a GPU where replayable faults are
not supported (Maxwell) it schedules onto an uninitialized queue and
dereferences NULL inside _raw_q_flush while holding the queue spinlock
with interrupts disabled. The suspend then cannot freeze user space and
the machine hangs with the display off.

Flush bottom_half_q only while replayable_faults.handling is true, the
same guard uvm_suspend() already uses for kill_channel_q.

Fixes: NVIDIA#1212
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kernel panic on resume from S3 suspend on Dell Precision 5510 (Quadro M1000M, nvidia_uvm NULL pointer dereference)

1 participant