uvm: Guard bottom-half queue flush on suspend - #1340
Open
tarik0 wants to merge 1 commit into
Open
Conversation
uvm_parent_gpu_init_isr() only creates isr.bottom_half_q when the GPU supports replayable faults. uvm_suspend() flushes that queue for every retained GPU without checking, so on a GPU where replayable faults are not supported (Maxwell) it schedules onto an uninitialized queue and dereferences NULL inside _raw_q_flush while holding the queue spinlock with interrupts disabled. The suspend then cannot freeze user space and the machine hangs with the display off. Flush bottom_half_q only while replayable_faults.handling is true, the same guard uvm_suspend() already uses for kill_channel_q. Fixes: NVIDIA#1212
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1212
Problem
On GPUs without replayable-fault support (Maxwell), suspending while any process holds
/dev/nvidia-uvmopen crashes the kernel and hangs the machine:The crashed task still holds the queue spinlock, so the suspend that follows fails to freeze user space and the system stays hung with the display already off. Reproduces 100% with a retained GPU; suspend only succeeds while no UVM client exists, which is why it looks intermittent.
Cause
uvm_parent_gpu_init_isr()createsisr.bottom_half_qonly insideif (parent_gpu->replayable_faults_supported), and setsisr.replayable_faults.handling = trueon the same path.uvm_maxwell.csetsreplayable_faults_supported = false, so the queue stays zero-initialised.uvm_suspend()flushes it for every retained GPU regardless, and_raw_q_schedule()doeslist_add_tail()on a NULL list head.Fix
Guard the flush with
isr.replayable_faults.handling, the flag set by the code that creates the queue.uvm_suspend()already guards thekill_channel_qflush on the next lines withnon_replayable_faults.handling; this makes the two consistent. Every GPU supported by the open kernel modules sets the flag, so their path is unchanged. The defect matters for the shared UVM source in the R580 proprietary driver, the last branch supporting Maxwell, Pascal and Volta. Please carry the fix into R580.Testing
nvidia-580xx-dkms, Arch/CachyOS)mem_sleep= deep (S3),NVreg_PreserveVideoMemoryAllocations=1,nvidia-suspend.serviceWithout the patch,
systemctl suspendwith a UVM client open produced the trace above and needed a hard reset (last occurrence in the same session, module build ID confirmed unpatched).With the patch applied to the DKMS source and the rebuilt module verified loaded by build ID: 4 suspend/resume cycles in one boot, all clean. Two with VS Code holding
/dev/nvidia-uvm; two with a CUDA driver-API program doing continuous work on a 16 MiBcuMemAllocManagedbuffer (PTX kernel increments every element once per second, host verifies each iteration). The program ran through both suspends without interruption: 195 iterations, 0 mismatches, values consistent across the sleep gaps. Journal per cycle:No
nvidia_uvmerrors, warnings or oopses in the kernel log for the whole boot. Tested on Maxwell only; no Turing-or-newer GPU available for a control run, but those GPUs already take the guarded path withhandling == true.Also reported on the forums: GM108M, 580.159.03, kernel 7.0, 545.29.02, kernel 6.6.