-
Notifications
You must be signed in to change notification settings - Fork 26.3k
[FSDP] Use reduce_scatter_tensor()
#87240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/87240
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit fe79b20: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
zhaojuanmao
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
somehow did not see it replaces the _reduce_scatter_base() call in fully_sharded_data_parallel.py?
After Olga's communication hook work, the reduce-scatter is now the one in @rohan-varma mentioned maybe the call is too non-obvious now. We can refactor later. |
rohan-varma
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. agree current situation is non-obvious, we should consider refactor.
|
@pytorchbot rebase -s |
|
@pytorchbot successfully started a rebase job. Check the current status here |
|
Rebase failed due to Command Raised by https://github.com/pytorch/pytorch/actions/runs/3309788180 |
Let us silence some more warnings 👍🏼 [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
Hey @awgu. |
Let us silence some more warnings 👍🏼 Pull Request resolved: pytorch#87240 Approved by: https://github.com/rohan-varma
Let us silence some more warnings 👍🏼 Pull Request resolved: pytorch#87240 Approved by: https://github.com/rohan-varma
Stack from ghstack:
reduce_scatter_tensor()#87240 [FSDP] Usereduce_scatter_tensor()Noneedge case #87308 [FSDP][2/N] Fix grad zero vs.Noneedge casesummon_full_params(with_grads)Nonegradient #87314 [FSDP][1/N] Updatesummon_full_params(with_grads)NonegradientLet us silence some more warnings 👍🏼