Skip to content

Enables configuration of NCCL communicators #97394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
syed-ahmed wants to merge 7 commits into from
Closed

Enables configuration of NCCL communicators #97394

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cd986f0
Adds ncclCommInitRankConfig features
syed-ahmed Mar 22, 2023
a86d4c7
Fixes lint and error
syed-ahmed Mar 24, 2023
5536c35
Fixes lint
syed-ahmed Mar 28, 2023
c45e6a2
Addresses review
syed-ahmed Apr 18, 2023
858f81d
Cleanup
syed-ahmed Apr 18, 2023
23959f1
Cleanup
syed-ahmed Apr 18, 2023
e6fa35b
Adds missing braces
syed-ahmed Apr 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixes lint
  • Loading branch information
@syed-ahmed
syed-ahmed committed May 15, 2023
commit 5536c357ea3e39be2007d01bb9b78aa5e41a8cc2
9 changes: 4 additions & 5 deletions torch/nn/parallel/distributed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -749,12 +749,11 @@ def __init__(
else:
self.process_group = process_group

if (
dist.get_backend(self.process_group) == "nccl"
and torch.cuda.nccl.version() >= (2, 17)
):
if dist.get_backend(
self.process_group
) == "nccl" and torch.cuda.nccl.version() >= (2, 17):
# Note: NVIDIA recommends using CGA Cluster Size of 2 when using DDP.
default_cga = dist.ProcessGroupNCCL.Options().config.cga_cluster_size # type: ignore[attr-defined]
default_cga = dist.ProcessGroupNCCL.Options().config.cga_cluster_size # type: ignore[attr-defined]
default_pg_nccl = self.process_group._get_backend(torch.device("cuda"))
current_cga = default_pg_nccl.options.config.cga_cluster_size
if current_cga == default_cga:
Expand Down