Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

eellison · 2022-10-04T23:27:21Z

Stack from ghstack (oldest at bottom):

-> Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and FakeTensorMode getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity.

… aot autograd [ghstack-poisoned]

pytorch-bot · 2022-10-04T23:27:23Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86245

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 1 Pending

As of commit 67dd731:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

… aot autograd ghstack-source-id: e59889b Pull Request resolved: #86245

torch/autograd/grad_mode.py

torch/csrc/autograd/init.cpp

ezyang · 2022-10-05T03:05:38Z

This looks basically fine but deferring to @albanD for final review.

albanD

Sounds good to me.

torch/csrc/autograd/init.cpp

torch/autograd/grad_mode.py

c10/core/AutogradState.h

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

albanD

Small error in doc, good to go otherwise!

albanD · 2022-10-05T17:51:59Z

docs/source/torch.rst

    is_grad_enabled
    inference_mode
-    is_inference_mode_enabled
+    set_multithreading_enabled


This should be reverted.

good catch, thanks

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

… aot autograd ghstack-source-id: c9480c0 Pull Request resolved: #86245

albanD

SGTM!

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

eellison · 2022-10-05T21:23:43Z

@pytorchbot rebase

pytorchmergebot · 2022-10-05T21:25:19Z

@pytorchbot successfully started a rebase job. Check the current status here

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

pytorchmergebot · 2022-10-05T21:25:32Z

Successfully rebased gh/eellison/335/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/86245)

… aot autograd ghstack-source-id: ba4a24c Pull Request resolved: #86245

eellison · 2022-10-06T03:26:10Z

@pytorchbot merge

pytorchmergebot · 2022-10-06T03:27:39Z

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered without a flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

github-actions · 2022-10-06T03:28:30Z

Hey @eellison.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

… aot autograd (#86245) (#86245) Summary: We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. Pull Request resolved: #86245 Approved by: https://github.com/albanD, https://github.com/yf225 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/d04889323e2bc0b7321b76e564292565c88b9a5e Reviewed By: seemethere Differential Revision: D40167028 Pulled By: seemethere fbshipit-source-id: f427c71e528deaa494521a61fcbf789d1a964711

Add Context Manager for Disabling Multithreading in Backwards, use in…

45fecc3

… aot autograd [ghstack-poisoned]

eellison requested review from albanD and soulitzer as code owners October 4, 2022 23:27

facebook-github-bot added the cla signed label Oct 4, 2022

eellison added a commit that referenced this pull request Oct 4, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

e8f0750

… aot autograd ghstack-source-id: e59889b Pull Request resolved: #86245

eellison requested review from Chillee and ezyang October 4, 2022 23:31

eellison commented Oct 4, 2022

View reviewed changes

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

ezyang reviewed Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

ezyang reviewed Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

albanD approved these changes Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

c10/core/AutogradState.h Show resolved Hide resolved

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 5, 2022

albanD reviewed Oct 5, 2022

View reviewed changes

eellison added a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

b2ca3a0

… aot autograd ghstack-source-id: c9480c0 Pull Request resolved: #86245

albanD approved these changes Oct 5, 2022

View reviewed changes

yanboliang mentioned this pull request Oct 5, 2022

TorchBench model tacotron2 failed due to assert device.type != "meta" pytorch/torchdynamo#1483

Closed

yf225 approved these changes Oct 5, 2022

View reviewed changes

eellison removed the ciflow/trunk Trigger trunk jobs on your pull request label Oct 5, 2022

pytorchmergebot pushed a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

bf1ecf2

… aot autograd ghstack-source-id: ba4a24c Pull Request resolved: #86245

pytorchmergebot added the Merged label Oct 6, 2022

pytorchmergebot closed this in d048893 Oct 6, 2022

eellison mentioned this pull request Oct 10, 2022

Turn on Data Dependent Throwing #86480

Closed

facebook-github-bot deleted the gh/eellison/335/head branch June 8, 2023 16:16

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

Uh oh!

Conversation

eellison commented Oct 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86245

✅ No Failures, 1 Pending

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezyang commented Oct 5, 2022

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

albanD Oct 5, 2022

Choose a reason for hiding this comment

Uh oh!

eellison Oct 5, 2022

Choose a reason for hiding this comment

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

eellison commented Oct 5, 2022

Uh oh!

pytorchmergebot commented Oct 5, 2022

Uh oh!

pytorchmergebot commented Oct 5, 2022

Uh oh!

eellison commented Oct 6, 2022

Uh oh!

pytorchmergebot commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

eellison commented Oct 4, 2022 •

edited

Loading

pytorch-bot bot commented Oct 4, 2022 •

edited

Loading