[CUDA graphs] Add warning if captured graph is empty #88754
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88754
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 57860ee: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
albanD
added
the
triaged
| TEST_WITH_ROCM or | ||
| int(torch.version.cuda.split(".")[0]) < 11, "CUDA >= 11.0 required for graphs") | ||
| def test_graph_warn_if_has_zero_nodes(self): | ||
| with warnings.catch_warnings(record=True) as caught: |
There was a problem hiding this comment.
The test is having issues with catching the warnings. @jjsjann123 pointed out that the possible cause is the call_guard in pybind11 https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/Graph.cpp#L31-L39
There was a problem hiding this comment.
You need the wrapper that clears the warning queue into Python. HANDLE_TH_ERRORS and END_HANDLE_TH_ERRORS_PYBIND check torch/csrc/README.md
cc @peterbell10
There was a problem hiding this comment.
I was going to suggest torch::wrap_pybind_function but it didn't work with member functions, so I've added that support and wrapped the CUDA graph functions in #88932. So, if you rebase it should start catching the warnings.
There was a problem hiding this comment.
Thanks @peterbell10,
The test is passing now!
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Fixes pytorch#87894 This PR adds a warning if captured graph is empty (consists of zero nodes). The example snippet where would it be useful: ```python import torch x = torch.randn(10) z = torch.zeros(10) g = torch.cuda.CUDAGraph() with torch.cuda.graph(g): z = x * x # Warn user ``` and in pytorch#87894 Pull Request resolved: pytorch#88754 Approved by: https://github.com/ezyang
6 participants
Fixes #87894
This PR adds a warning if captured graph is empty (consists of zero nodes).
The example snippet where would it be useful:
and in #87894