Skip to content

[14/N] Refactor _new_process_group_helper() to remove repeated code #88351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
H-Huang wants to merge 4 commits into from
Closed

[14/N] Refactor _new_process_group_helper() to remove repeated code #88351

H-Huang wants to merge 4 commits into from

Conversation

@H-Huang
Copy link
Member

@H-Huang H-Huang commented Nov 2, 2022 •
edited
Loading

Stack from ghstack:

Changes

  • refactor parts of _new_process_group_helper() to remove repeated code

Context

#86225

Differential Revision: D41188274

@H-Huang H-Huang mentioned this pull request Nov 2, 2022
Closed
@pytorch-bot pytorch-bot bot added the release notes: distributed (c10d) release notes category label Nov 2, 2022
@H-Huang H-Huang mentioned this pull request Nov 2, 2022
Closed
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 2, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88351

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a73a1c5:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 2, 2022
Closed
Closed
@H-Huang
Update on "[14/N] Refactor _new_process_group_helper() to remove repe…
10e4c14 
…ated code"


Changes:
- refactor parts of `_new_process_group_helper()` to remove repeated code

[ghstack-poisoned]
@H-Huang H-Huang mentioned this pull request Nov 4, 2022
Closed
kwen2501
kwen2501 approved these changes Nov 7, 2022
View reviewed changes
Copy link
Collaborator

@kwen2501 kwen2501 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for refactoring.
Just added minor comments.

torch/distributed/distributed_c10d.py Outdated
if backend in [Backend.GLOO, Backend.NCCL, Backend.UCC]:
# In debug mode and if GLOO is available, wrap in a wrapper PG that
# enables enhanced collective checking for debugability.
if get_debug_level() == DebugLevel.DETAIL:
Copy link
Collaborator

@kwen2501 kwen2501 Nov 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if condition should be put out-most.

Copy link
Member Author

@H-Huang H-Huang Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure I can move it outmost, looks a bit cleaner that way

torch/distributed/distributed_c10d.py Outdated
GLOO is not available. Build with Gloo to
create a wrapper process group in debug mode
to aid collective desynchronization debugging."""
)
Copy link
Collaborator

@kwen2501 kwen2501 Nov 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move the current out-most condition as an elif here:

elif backend not in [Backend.GLOO, Backend.NCCL, Backend.UCC]:
  logger.info(
    """TORCH_DISTRIBUTED_DEBUG was set to DETAIL, but the current backend does not support this feature, ignoring."""
  )
else:

Copy link
Member Author

@H-Huang H-Huang Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we want to add a log for this? We currently don't have a message. I am also not sure if there are things other than ProcessGroupWrapper that is available when dist.DebugLevel.DETAIL is set, maybe the logs are also more verbose so maybe we should just say the current backend does not support ProcessGroupWrapper?

@H-Huang
Update on "[14/N] Refactor _new_process_group_helper() to remove repe…
cc2fcd8 
…ated code"


Changes:
- refactor parts of `_new_process_group_helper()` to remove repeated code

[ghstack-poisoned]
awgu
awgu reviewed Nov 9, 2022
View reviewed changes
Copy link
Collaborator

@awgu awgu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing commits as well I see...

torch/distributed/distributed_c10d.py Outdated
# Process group wrapper initialization for supported PGs when TORCH_DISTRIBUTED_DEBUG is set
if backend in [Backend.GLOO, Backend.NCCL, Backend.UCC]:
# In debug mode and if GLOO is available, wrap in a wrapper PG that
# enables enhanced collective checking for debugability.
Copy link
Collaborator

@awgu awgu Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
# enables enhanced collective checking for debugability.
# enables enhanced collective checking for debuggability.

https://wikidiff.com/debugability/debuggability

Copy link
Member Author

@H-Huang H-Huang Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you sir

@H-Huang
Update on "[14/N] Refactor _new_process_group_helper() to remove repe…
a73a1c5 
…ated code"


Changes:
- refactor parts of `_new_process_group_helper()` to remove repeated code

[ghstack-poisoned]
@H-Huang H-Huang added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 10, 2022
@H-Huang
Copy link
Member Author

H-Huang commented Nov 10, 2022

@H-Huang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@H-Huang
Copy link
Member Author

H-Huang commented Nov 10, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 10, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

This was referenced Nov 10, 2022
Closed
Closed
@H-Huang
Copy link
Member Author

H-Huang commented Nov 11, 2022

@H-Huang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@H-Huang H-Huang mentioned this pull request Nov 11, 2022
Closed
kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Dec 10, 2022
@H-Huang @kulinseth
[14/N] Refactor _new_process_group_helper() to remove repeated code (p…
d9e7fdf 
…ytorch#88351)

Changes:
- refactor parts of `_new_process_group_helper()` to remove repeated code

Differential Revision: [D41188274](https://our.internmc.facebook.com/intern/diff/D41188274)
Pull Request resolved: pytorch#88351
Approved by: https://github.com/kwen2501
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@awgu awgu awgu left review comments

@kwen2501 kwen2501 kwen2501 approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (c10d) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@H-Huang @pytorchmergebot @kwen2501 @awgu