Print all traceback for nested backwards in detect_anomaly #43626

mfkasim1 · 2020-08-26T15:55:17Z

This pull request adds a feature of printing all tracebacks if a detect_anomaly mode detects nan in nested backward operations.
The way I did it is by assigning a node as a parent to all nodes it produces during its backward calculation. Then if one of the children produces nan, it will print the traceback from the parent and grand parents (if any).

The parent is assigned in parent_node_ member in Node class which is accessible in C++ by function node->parent() and in Python by node.parent_function.
A node has a parent iff:

it is created from a backward operation, and
created when anomaly mode and grad mode are both enabled.

An example of this feature:

import torch

def example():
    x = torch.tensor(1.0, requires_grad=True)
    y = torch.tensor(1e-8, requires_grad=True)  # small to induce nan in n-th backward
    a = x * y
    b = x * y
    z1 = a / b  # can produce nan in n-th backward as long as #43414 is unsolved
    z = z1 * z1
    gy , = torch.autograd.grad( z , (y,), create_graph=True)
    gy2, = torch.autograd.grad(gy , (y,), create_graph=True)
    gy3, = torch.autograd.grad(gy2, (y,), create_graph=True)
    gy4, = torch.autograd.grad(gy3, (y,), create_graph=True)
    return gy4

with torch.autograd.detect_anomaly():
    gy4 = example()

with output:

example.py:16: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
  with torch.autograd.detect_anomaly():
/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py:190: UserWarning: Error detected in DivBackward0. Traceback of forward call that caused the error:
  File "example.py", line 17, in <module>
    gy4 = example()
  File "example.py", line 12, in example
    gy3, = torch.autograd.grad(gy2, (y,), create_graph=True)
  File "/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 190, in grad
    return Variable._execution_engine.run_backward(
 (Triggered internally at  ../torch/csrc/autograd/python_anomaly_mode.cpp:61.)
  return Variable._execution_engine.run_backward(
/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py:190: UserWarning:

Traceback of forward call that induces the previous calculation:
  File "example.py", line 17, in <module>
    gy4 = example()
  File "example.py", line 11, in example
    gy2, = torch.autograd.grad(gy , (y,), create_graph=True)
  File "/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 190, in grad
    return Variable._execution_engine.run_backward(
 (Triggered internally at  ../torch/csrc/autograd/python_anomaly_mode.cpp:65.)
  return Variable._execution_engine.run_backward(
/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py:190: UserWarning:

Traceback of forward call that induces the previous calculation:
  File "example.py", line 17, in <module>
    gy4 = example()
  File "example.py", line 8, in example
    z1 = a / b  # can produce nan in n-th backward as long as #43414 is unsolved
 (Triggered internally at  ../torch/csrc/autograd/python_anomaly_mode.cpp:65.)
  return Variable._execution_engine.run_backward(
Traceback (most recent call last):
  File "example.py", line 17, in <module>
    gy4 = example()
  File "example.py", line 13, in example
    gy4, = torch.autograd.grad(gy3, (y,), create_graph=True)
  File "/home/mfkasim/anaconda2/envs/base3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 190, in grad
    return Variable._execution_engine.run_backward(
RuntimeError: Function 'DivBackward0' returned nan values in its 1th output.

cc & thanks to @albanD

dr-ci · 2020-08-26T16:06:15Z

💊 CI failures summary and remediations

As of commit 46b57a1 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 21 times.

albanD

Thanks for the PR.
The high level idea looks good.
I think we can simplify a bit the implementation though. Can you check the inline comments below and let me know what you think.

torch/csrc/autograd/function.h

torch/csrc/autograd/engine.cpp

albanD

This looks quite good, a few more comments below.

torch/csrc/autograd/function.cpp

torch/csrc/autograd/function.h

torch/csrc/autograd/python_anomaly_mode.cpp

torch/csrc/autograd/function.h

torch/csrc/autograd/anomaly_mode.h

albanD

the c++ looks good!
Just had a chance to take a quick look at the tests. They are good, just a small comment to make sure things are tested and we don't leak memory.
Then it should be good to go!

test/test_autograd.py

torch/csrc/autograd/python_anomaly_mode.cpp

torch/csrc/autograd/function.cpp

albanD

LGTM

Can you just edit the comment mentioned above and this will be good to go!

facebook-github-bot

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

mfkasim1 · 2020-08-29T06:58:27Z

Thanks for your help @albanD!

facebook-github-bot · 2020-08-31T16:11:21Z

@albanD merged this pull request in 576880f.

mfkasim1 added 3 commits August 26, 2020 14:38

Print all traceback for nested backward in detect_anomaly

dcf1807

Add a test to detect the printed traceback for nested backward

7b1fee6

Complying with the writing standard

0011ee0

mfkasim1 requested review from albanD and apaszke as code owners August 26, 2020 15:55

pytorchbot added the open source label Aug 26, 2020

albanD reviewed Aug 26, 2020

View reviewed changes

torch/csrc/autograd/function.h Outdated Show resolved Hide resolved

torch/csrc/autograd/function.h Outdated Show resolved Hide resolved

torch/csrc/autograd/engine.cpp Outdated Show resolved Hide resolved

Moved parent to metadata

bb70596

mfkasim1 requested a review from albanD August 27, 2020 10:48

albanD reviewed Aug 27, 2020

View reviewed changes

mfkasim1 added 2 commits August 27, 2020 18:38

Various refactoring

dc4632a

Removed outdated comments

225f222

mfkasim1 requested a review from albanD August 27, 2020 18:22

albanD reviewed Aug 27, 2020

View reviewed changes

test/test_autograd.py Show resolved Hide resolved

test/test_autograd.py Outdated Show resolved Hide resolved

torch/csrc/autograd/python_anomaly_mode.cpp Outdated Show resolved Hide resolved

Fixing the test for nested backward anomaly and fix the bug discovered

278dac3

mfkasim1 commented Aug 27, 2020

View reviewed changes

torch/csrc/autograd/function.cpp Outdated Show resolved Hide resolved

albanD approved these changes Aug 28, 2020

View reviewed changes

facebook-github-bot reviewed Aug 28, 2020

View reviewed changes

Removed unnecessary comments

46b57a1

facebook-github-bot reviewed Aug 28, 2020

View reviewed changes

facebook-github-bot closed this in 576880f Aug 31, 2020

facebook-github-bot added the merged label Aug 31, 2020

mruberry added the Merged label Oct 28, 2020

albanD mentioned this pull request Jan 30, 2021

autograd.grad with set_detect_anomaly(True) will cause memory leak #51349

Closed

Print all traceback for nested backwards in detect_anomaly #43626

Print all traceback for nested backwards in detect_anomaly #43626

Uh oh!

Conversation

mfkasim1 commented Aug 26, 2020

Uh oh!

dr-ci bot commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

mfkasim1 commented Aug 29, 2020

Uh oh!

facebook-github-bot commented Aug 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dr-ci bot commented Aug 26, 2020 •

edited

Loading

albanD left a comment •

edited

Loading