Skip to content

[FSDP][optim_state_dict][9/N] Rewrite the all-gather flow of optimizer state to support older GPUs #91343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fegin wants to merge 9 commits into from
Closed

[FSDP][optim_state_dict][9/N] Rewrite the all-gather flow of optimizer state to support older GPUs #91343

fegin wants to merge 9 commits into from

Conversation

@fegin
Copy link
Contributor

@fegin fegin commented Dec 23, 2022 •
edited
Loading

@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
4d18cda 
…re all_gather_object work correctly on older GPUs

[ghstack-poisoned]
@fegin fegin mentioned this pull request Dec 23, 2022
Closed
@pytorch-bot pytorch-bot bot added the release notes: distributed (fsdp) release notes category label Dec 23, 2022
@pytorch-bot
Copy link

pytorch-bot bot commented Dec 23, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91343

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6d2d257:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

fegin added a commit that referenced this pull request Dec 23, 2022
@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
62efcdf 
…re all_gather_object work correctly on older GPUs

ghstack-source-id: 1c77b63
Pull Request resolved: #91343
@fegin fegin marked this pull request as draft December 23, 2022 01:15
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
a380ad7 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
5e74382 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
fegin added a commit that referenced this pull request Dec 29, 2022
@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
2dca88f 
…re all_gather_object work correctly on older GPUs

ghstack-source-id: 9bd47e7
Pull Request resolved: #91343
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
263c443 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
2836049 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
0feefd6 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
@fegin
Update on "[FSDP][optim_state_dict][9/N] Specially treat zero dim ten…
d262661 
…sor to ensure all_gather_object work correctly on older GPUs"

[ghstack-poisoned]
fegin added a commit that referenced this pull request Jan 12, 2023
@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
548c5c3 
…re all_gather_object work correctly on older GPUs

ghstack-source-id: 940633e
Pull Request resolved: #91343
@fegin fegin added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 13, 2023
@fegin fegin changed the title [FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensure all_gather_object work correctly on older GPUs [FSDP][optim_state_dict][9/N] Rewrite the all-gather flow of optimizer state to support older GPUs Jan 13, 2023
rohan-varma
rohan-varma approved these changes Jan 13, 2023
View reviewed changes
Copy link
Contributor

@rohan-varma rohan-varma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for ensuring this is covered by CI!

@fegin fegin marked this pull request as ready for review January 13, 2023 23:36
@fegin
Update on "[FSDP][optim_state_dict][9/N] Rewrite the all-gather flow …
14e7993 
…of optimizer state to support older GPUs"

[ghstack-poisoned]
fegin added a commit that referenced this pull request Jan 13, 2023
@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
386fe42 
…re all_gather_object work correctly on older GPUs

ghstack-source-id: c472cee
Pull Request resolved: #91343
@fegin
Update on "[FSDP][optim_state_dict][9/N] Rewrite the all-gather flow …
6d2d257 
…of optimizer state to support older GPUs"

[ghstack-poisoned]
fegin added a commit that referenced this pull request Jan 17, 2023
@fegin
[FSDP][optim_state_dict][9/N] Specially treat zero dim tensor to ensu…
9addd26 
…re all_gather_object work correctly on older GPUs

ghstack-source-id: 5455ea9
Pull Request resolved: #91343
@fegin
Copy link
Contributor Author

fegin commented Jan 17, 2023

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jan 17, 2023

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jan 17, 2023

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / win-vs2019-cuda11.6-py3 / build

Details for Dev Infra team Raised by workflow job

@fegin
Copy link
Contributor Author

fegin commented Jan 17, 2023

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jan 17, 2023

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@facebook-github-bot facebook-github-bot deleted the gh/fegin/58/head branch June 8, 2023 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@wanchaol wanchaol Awaiting requested review from wanchaol

1 more reviewer

@rohan-varma rohan-varma rohan-varma approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (fsdp) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fegin @pytorchmergebot @rohan-varma