[Reland][FSDP] Mixed precision enablement" #75024

rohan-varma · 2022-03-31T16:55:40Z

Stack from ghstack (oldest at bottom):

Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions.

Original commit changeset: 99295ea4ff02

Original Phabricator Diff: D35000703

Differential Revision: D35287501

Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) [ghstack-poisoned]

facebook-github-bot · 2022-03-31T16:55:43Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/75024
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
❓Need help or want to give feedback on the CI? Visit our office hours

💊 CI failures summary and remediations

As of commit 01e3c46 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-05T14:29:19.6988886Z The PR is introduc...m to confirm whether this change is wanted or not.

2022-04-05T14:29:19.6976754Z processing existing schema:  text(__torch__.torch.classes.profiling.SourceRef _0) -> (str _0)
2022-04-05T14:29:19.6978200Z processing existing schema:  count(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-05T14:29:19.6979857Z processing existing schema:  duration_ns(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-05T14:29:19.6980285Z processing existing schema:  source(__torch__.torch.classes.profiling.SourceStats _0) -> (__torch__.torch.classes.profiling.SourceRef _0)
2022-04-05T14:29:19.6981907Z processing existing schema:  line_map(__torch__.torch.classes.profiling.SourceStats _0) -> (Dict(int, __torch__.torch.classes.profiling.InstructionStats) _0)
2022-04-05T14:29:19.6982784Z processing existing schema:  __init__(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-05T14:29:19.6984127Z processing existing schema:  enable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-05T14:29:19.6985303Z processing existing schema:  disable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-05T14:29:19.6987031Z processing existing schema:  _dump_stats(__torch__.torch.classes.profiling._ScriptProfile _0) -> (__torch__.torch.classes.profiling.SourceStats[] _0)
2022-04-05T14:29:19.6988535Z processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (NoneType _0)
2022-04-05T14:29:19.6988886Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2022-04-05T14:29:19.6988900Z 
2022-04-05T14:29:19.6988977Z Broken ops: [
2022-04-05T14:29:19.6989224Z 	quantized::softmax(Tensor qx, int dim, float output_scale, int output_zero_point) -> (Tensor)
2022-04-05T14:29:19.6989289Z ]
2022-04-05T14:29:19.8019105Z + cleanup
2022-04-05T14:29:19.8019519Z + retcode=1
2022-04-05T14:29:19.8019589Z + set +x
2022-04-05T14:29:19.8059390Z ##[error]Process completed with exit code 1.
2022-04-05T14:29:19.8106133Z Prepare all required actions
2022-04-05T14:29:19.8106310Z Getting action download info

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) ghstack-source-id: 152704893 Pull Request resolved: #75024

mrshenli

Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions.

Do we need to mention this in our docs?

Reland #74452 Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions. Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) [ghstack-poisoned]

rohan-varma · 2022-04-04T20:03:15Z

Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions.

Do we need to mention this in our docs?

Yep, we should definitely add this. Filed #75215

Reland #74452 Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions. Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) [ghstack-poisoned]

Pull Request resolved: #75024 Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 ghstack-source-id: 152989929 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/)

Reland #74452 Issue was older nccl version does not support bf16. Will take an approach similar to #67843 to ensure test only runs with later nccl versions. Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) [ghstack-poisoned]

…75024) Summary: Pull Request resolved: #75024 Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 (6b0b088) ghstack-source-id: 153059190 (Note: this ignores all push blocking failures!) Test Plan: CI Reviewed By: pbelevich Differential Revision: D35287501 fbshipit-source-id: c6c9ada039de27cf9cc477561f92a7f888bdf5f7

github-actions · 2022-04-05T21:16:33Z

Hey @rohan-varma.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Back out "Revert D35000703: [WIP][FSDP] Mixed precision enablement"

6925769

Original commit changeset: 99295ea4ff02 Original Phabricator Diff: D35000703 Differential Revision: [D35287501](https://our.internmc.facebook.com/intern/diff/D35287501/) [ghstack-poisoned]

rohan-varma requested review from H-Huang, awgu, mingzhe09088, mrshenli, pritamdamania87 and zhaojuanmao as code owners March 31, 2022 16:55

facebook-github-bot added cla signed oncall: distributed Add this issue/PR to distributed oncall triage queue labels Mar 31, 2022

rohan-varma changed the title ~~Back out "Revert D35000703: [WIP][FSDP] Mixed precision enablement"~~ [Reland][FSDP] Mixed precision enablement" Mar 31, 2022

mrshenli approved these changes Mar 31, 2022

View reviewed changes

rohan-varma added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 4, 2022

rohan-varma added ciflow/trunk Trigger trunk jobs on your pull request and removed ciflow/trunk Trigger trunk jobs on your pull request labels Apr 4, 2022

This was referenced Apr 4, 2022

[FSDP] Code simplification #75227

Closed

[FSDP] Enhance test for checkpoint #75236

Closed

pytorchmergebot closed this in 3a0b393 Apr 5, 2022

rohan-varma mentioned this pull request Apr 8, 2022

Make FSDP Mixed Precision work with BatchNorm #75478

Closed

facebook-github-bot deleted the gh/rohan-varma/530/head branch April 8, 2022 23:01

rohan-varma mentioned this pull request May 3, 2022

support FSDP with AMP #76607

Closed

WBobby mentioned this pull request Aug 17, 2022

Add ROCm5.2.3/AMDGPU support for PyTorch WBobby/pytorch#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Reland][FSDP] Mixed precision enablement" #75024

[Reland][FSDP] Mixed precision enablement" #75024

Uh oh!

rohan-varma commented Mar 31, 2022 •

edited

Loading

Uh oh!

facebook-github-bot commented Mar 31, 2022 •

edited

Loading

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Uh oh!

mrshenli left a comment

Uh oh!

rohan-varma commented Apr 4, 2022

Uh oh!

github-actions bot commented Apr 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[Reland][FSDP] Mixed precision enablement" #75024

[Reland][FSDP] Mixed precision enablement" #75024

Uh oh!

Conversation

rohan-varma commented Mar 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Mar 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Uh oh!

mrshenli left a comment

Choose a reason for hiding this comment

Uh oh!

rohan-varma commented Apr 4, 2022

Uh oh!

github-actions bot commented Apr 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rohan-varma commented Mar 31, 2022 •

edited

Loading

facebook-github-bot commented Mar 31, 2022 •

edited

Loading