Add env variable to bypass CUDACachingAllocator for debugging #45294
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: While tracking down a recent memory corruption bug we found that cuda-memcheck wasn't finding the bad accesses, and @ngimel pointed out that it's because we use a caching allocator so a lot of "out of bounds" accesses land in a valid slab. This PR adds a runtime knob (`PYTORCH_CUDA_DEBUG_MEMORY`) that, when set, bypasses the caching allocator's caching logic so that allocations go straight to cudaMalloc. This way, cuda-memcheck will actually work. Test Plan: Insert some memory errors and run a test under cuda-memcheck; observe that cuda-memcheck flags an error where expected. Specifically I removed the output-masking logic here: https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tensorexpr/cuda_codegen.cpp#L819-L826 And ran: ``` PYTORCH_CUDA_DEBUG_MEMORY=1 cuda-memcheck pytest -k test_superslomo test_jit_fuser_te.py ``` [ghstack-poisoned]
bertmaher
added a commit
that referenced
this pull request
Sep 24, 2020
Summary: While tracking down a recent memory corruption bug we found that cuda-memcheck wasn't finding the bad accesses, and @ngimel pointed out that it's because we use a caching allocator so a lot of "out of bounds" accesses land in a valid slab. This PR adds a runtime knob (`PYTORCH_CUDA_DEBUG_MEMORY`) that, when set, bypasses the caching allocator's caching logic so that allocations go straight to cudaMalloc. This way, cuda-memcheck will actually work. Test Plan: Insert some memory errors and run a test under cuda-memcheck; observe that cuda-memcheck flags an error where expected. Specifically I removed the output-masking logic here: https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tensorexpr/cuda_codegen.cpp#L819-L826 And ran: ``` PYTORCH_CUDA_DEBUG_MEMORY=1 cuda-memcheck pytest -k test_superslomo test_jit_fuser_te.py ``` ghstack-source-id: 5a28c87 Pull Request resolved: #45294
Contributor
Author
|
So this is a pretty minimal debug path for the cuda allocator; I'd love some advice on whether this is a reasonable approach, and whether there's anything else I should be doing here. Also, suggestions for perf testing would be great -- I think we have an operator overhead bench that I'll try out, anything else? It seems like a perf-sensitive path so with any luck it doesn't slow that down. |
Codecov Report
@@ Coverage Diff @@
## gh/bertmaher/23/base #45294 +/- ##
======================================================
Coverage 68.05% 68.05%
======================================================
Files 396 393 -3
Lines 51232 50914 -318
======================================================
- Hits 34864 34651 -213
+ Misses 16368 16263 -105
Continue to review full report at Codecov.
|
dzhulgakov
approved these changes
Sep 25, 2020
Collaborator
dzhulgakov
left a comment
dzhulgakov
left a comment
There was a problem hiding this comment.
Looks good. Maybe call it PYTORCH_NO_CUDA_MEMORY_CACHING or something like that? 'debugging' implies some fancy tool/report
Contributor
|
Yeah, this looks fine, but agreed with Dmytro on env var renaming. You should also think about where to document this option. |
Collaborator
|
This section https://pytorch.org/docs/master/cuda.html#memory-management looks like a reasonable place for documentation |
…ing" Summary: While tracking down a recent memory corruption bug we found that cuda-memcheck wasn't finding the bad accesses, and @ngimel pointed out that it's because we use a caching allocator so a lot of "out of bounds" accesses land in a valid slab. This PR adds a runtime knob (`PYTORCH_CUDA_DEBUG_MEMORY`) that, when set, bypasses the caching allocator's caching logic so that allocations go straight to cudaMalloc. This way, cuda-memcheck will actually work. Test Plan: Insert some memory errors and run a test under cuda-memcheck; observe that cuda-memcheck flags an error where expected. Specifically I removed the output-masking logic here: https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tensorexpr/cuda_codegen.cpp#L819-L826 And ran: ``` PYTORCH_CUDA_DEBUG_MEMORY=1 cuda-memcheck pytest -k test_superslomo test_jit_fuser_te.py ``` [ghstack-poisoned]
bertmaher
added a commit
that referenced
this pull request
Sep 28, 2020
Summary: While tracking down a recent memory corruption bug we found that cuda-memcheck wasn't finding the bad accesses, and @ngimel pointed out that it's because we use a caching allocator so a lot of "out of bounds" accesses land in a valid slab. This PR adds a runtime knob (`PYTORCH_NO_CUDA_MEMORY_CACHING`) that, when set, bypasses the caching allocator's caching logic so that allocations go straight to cudaMalloc. This way, cuda-memcheck will actually work. Test Plan: Insert some memory errors and run a test under cuda-memcheck; observe that cuda-memcheck flags an error where expected. Specifically I removed the output-masking logic here: https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tensorexpr/cuda_codegen.cpp#L819-L826 And ran: ``` PYTORCH_NO_CUDA_MEMORY_CACHING=1 cuda-memcheck pytest -k test_superslomo test_jit_fuser_te.py ``` ghstack-source-id: 6b44289 Pull Request resolved: #45294
Contributor
|
@bertmaher merged this pull request in 03342af. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary: While tracking down a recent memory corruption bug we found that
cuda-memcheck wasn't finding the bad accesses, and @ngimel pointed out that
it's because we use a caching allocator so a lot of "out of bounds" accesses
land in a valid slab.
This PR adds a runtime knob (
PYTORCH_CUDA_DEBUG_MEMORY) that, when set,bypasses the caching allocator's caching logic so that allocations go straight
to cudaMalloc. This way, cuda-memcheck will actually work.
Test Plan: Insert some memory errors and run a test under cuda-memcheck;
observe that cuda-memcheck flags an error where expected.
Specifically I removed the output-masking logic here:
https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tensorexpr/cuda_codegen.cpp#L819-L826
And ran:
Differential Revision: D23964734