Skip to content

[FSDP] Allow MixedPrecision to skip inputs #90620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrshenli wants to merge 6 commits into from
Closed

[FSDP] Allow MixedPrecision to skip inputs #90620

mrshenli wants to merge 6 commits into from

Conversation

@mrshenli
Copy link
Contributor

@mrshenli mrshenli commented Dec 10, 2022 •
edited
Loading

@mrshenli mrshenli mentioned this pull request Dec 10, 2022
Closed
@pytorch-bot
Copy link

pytorch-bot bot commented Dec 10, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90620

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 85a007b:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: distributed (fsdp) release notes category label Dec 10, 2022
mrshenli added a commit that referenced this pull request Dec 10, 2022
@mrshenli
[FSDP] Allos MixedPrecision to skip inputs
9bf2c3c 
ghstack-source-id: a6d90b9
Pull Request resolved: #90620
@mrshenli mrshenli mentioned this pull request Dec 10, 2022
Closed
rohan-varma
rohan-varma approved these changes Dec 10, 2022
View reviewed changes
Copy link
Contributor

@rohan-varma rohan-varma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but curious what's the use case to skip inputs?

@mrshenli
Copy link
Contributor Author

mrshenli commented Dec 10, 2022

Thanks @rohan-varma, one model I am working on has one forward argument, which is sensitive to precision. So have to keep it in fp32, instead of converting it to bfloat16 and convert back.

@mrshenli mrshenli added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 10, 2022
awgu
awgu reviewed Dec 10, 2022
View reviewed changes
@awgu awgu changed the title [FSDP] Allos MixedPrecision to skip inputs [FSDP] Allow MixedPrecision to skip inputs Dec 10, 2022
awgu
awgu reviewed Dec 10, 2022
View reviewed changes
awgu
awgu reviewed Dec 10, 2022
View reviewed changes
awgu
awgu reviewed Dec 10, 2022
View reviewed changes
awgu
awgu approved these changes Dec 10, 2022
View reviewed changes
Copy link
Collaborator

@awgu awgu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me. Sorry for leaving all of the comments in separate reviews instead of just one.

@awgu
Copy link
Collaborator

awgu commented Dec 11, 2022

My bad for suggesting to un-default cast_forward_inputs. I did not see that it already existed in past tests.

I think you need to add a value for cast_forward_inputs in this line:

pytorch/test/distributed/fsdp/test_fsdp_mixed_precision.py

Line 821 in a5d34ad

model = SaveForwardInputsModel(forward_inputs).cuda()

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Dec 11, 2022

This pull request has been merged in 80542ad.

mrshenli added a commit to mrshenli/pytorch that referenced this pull request Dec 12, 2022
@mrshenli
[FSDP] Allow MixedPrecision to skip inputs
b65e8ec 
ghstack-source-id: d9e229f
Pull Request resolved: pytorch#90620
@mrshenli mrshenli mentioned this pull request Dec 13, 2022
Closed
@facebook-github-bot facebook-github-bot deleted the gh/mrshenli/355/head branch June 8, 2023 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@awgu awgu awgu approved these changes

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@H-Huang H-Huang Awaiting requested review from H-Huang

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@wanchaol wanchaol Awaiting requested review from wanchaol

+1 more reviewer

@rohan-varma rohan-varma rohan-varma approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (fsdp) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrshenli @awgu @facebook-github-bot @rohan-varma