Skip to content

[9/N] [Dispatchable Collectives] Update reduce_scatter with CPU / CUDA implementations #86166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
H-Huang wants to merge 5 commits into from
Closed

[9/N] [Dispatchable Collectives] Update reduce_scatter with CPU / CUDA implementations #86166

H-Huang wants to merge 5 commits into from

Conversation

@H-Huang
Copy link
Member

@H-Huang H-Huang commented Oct 3, 2022 •
edited
Loading

Stack from ghstack:

Changes

  • Updates for the reduce_scatter collective

Context

#86225

This was referenced Oct 3, 2022
Closed
Closed
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 3, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86166

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 6 Pending

As of commit 569b2f7:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@H-Huang H-Huang mentioned this pull request Oct 3, 2022
Closed
@pytorch-bot pytorch-bot bot added the release notes: distributed (c10d) release notes category label Oct 3, 2022
@H-Huang H-Huang mentioned this pull request Oct 3, 2022
Closed
H-Huang added a commit that referenced this pull request Oct 3, 2022
@H-Huang
[9/N] [Dispatchable Collectives] Update reduce_scatter with CPU / CUD…
0c5bc9e 
…A implementations

ghstack-source-id: ea9c95f
Pull Request resolved: #86166
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Oct 3, 2022

/easycla

As part of the transition to the PyTorch Foundation, this project now requires contributions be covered under the new CLA. See #85559 for additional details.

This comment will trigger a new check of this PR. If you are already covered, you will simply see a new "EasyCLA" check that passes. If you are not covered, a bot will leave a new comment with a link to sign.

@H-Huang
Update on "[9/N] [Dispatchable Collectives] Update reduce_scatter wit…
f6c9c82 
…h CPU / CUDA implementations"


### Changes
- Updates for the reduce_scatter collective

### Context
#86225


[ghstack-poisoned]
@H-Huang H-Huang mentioned this pull request Oct 6, 2022
Closed
@H-Huang
Copy link
Member Author

H-Huang commented Oct 6, 2022

@H-Huang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@H-Huang
Update on "[9/N] [Dispatchable Collectives] Update reduce_scatter wit…
f9771a9 
…h CPU / CUDA implementations"


### Changes
- Updates for the reduce_scatter collective

### Context
#86225

Differential Revision: [D40145699](https://our.internmc.facebook.com/intern/diff/D40145699)

[ghstack-poisoned]
This was referenced Oct 6, 2022
Closed
Closed
Closed
wanchaol
wanchaol reviewed Oct 6, 2022
View reviewed changes
test/distributed/test_c10d_common.py
if collective == dist.all_gather:
collective([tensor], tensor, *args)
elif collective == dist.reduce_scatter:
if backend != "gloo":
Copy link
Collaborator

@wanchaol wanchaol Oct 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the dispatchable functionality, what would happen if user call reduce_scatter on CPU?

Copy link
Member Author

@H-Huang H-Huang Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed this question. It will call the reduce_scatter based on the configured backend for CPU (we will set a backend for CPU by default, but users can configure this). This is yet to be implemented in this PR

@H-Huang
Update on "[9/N] [Dispatchable Collectives] Update reduce_scatter wit…
a8b9265 
…h CPU / CUDA implementations"


### Changes
- Updates for the reduce_scatter collective

### Context
#86225

Differential Revision: [D40145699](https://our.internmc.facebook.com/intern/diff/D40145699)

[ghstack-poisoned]
@H-Huang
Copy link
Member Author

H-Huang commented Oct 7, 2022

@H-Huang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@H-Huang H-Huang mentioned this pull request Oct 17, 2022
Closed
kwen2501
kwen2501 approved these changes Oct 28, 2022
View reviewed changes
Copy link
Collaborator

@kwen2501 kwen2501 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 28, 2022
@H-Huang
Update on "[9/N] [Dispatchable Collectives] Update reduce_scatter wit…
569b2f7 
…h CPU / CUDA implementations"


### Changes
- Updates for the reduce_scatter collective

### Context
#86225

[ghstack-poisoned]
@H-Huang
Copy link
Member Author

H-Huang commented Nov 1, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 1, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

This was referenced Nov 2, 2022
Closed
Closed
kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Nov 5, 2022
@H-Huang @kulinseth
[9/N] [Dispatchable Collectives] Update reduce_scatter with CPU / CUD…
3f95020 
…A implementations (pytorch#86166)

### Changes
- Updates for the reduce_scatter collective

### Context
pytorch#86225
Pull Request resolved: pytorch#86166
Approved by: https://github.com/kwen2501
kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Dec 10, 2022
@H-Huang @kulinseth
[9/N] [Dispatchable Collectives] Update reduce_scatter with CPU / CUD…
0e99a0e 
…A implementations (pytorch#86166)

### Changes
- Updates for the reduce_scatter collective

### Context
pytorch#86225
Pull Request resolved: pytorch#86166
Approved by: https://github.com/kwen2501
@facebook-github-bot facebook-github-bot deleted the gh/H-Huang/83/head branch June 8, 2023 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wanchaol wanchaol wanchaol left review comments

@kwen2501 kwen2501 kwen2501 approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@awgu awgu Awaiting requested review from awgu

@mingzhe09088 mingzhe09088 Awaiting requested review from mingzhe09088

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request cla signed Merged release notes: distributed (c10d) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@H-Huang @facebook-github-bot @pytorchmergebot @kwen2501 @wanchaol