Update DDP docs for Dynamo/DDPOptimizer #89096
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/89096
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 FailuresAs of commit 55d73a6: The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| .. code:: | ||
| ddp_model = DDP(model, device_ids=[rank]) | ||
| ddp_model = torch.compile(ddp_model) |
There was a problem hiding this comment.
So we can't merge this quite yet since won't exist until sometime next week, in the meantime you can can use the optimize API if you'd rather merge this now
There was a problem hiding this comment.
no rush, i can just land it once its ready. keep me posted.
| ------------------------ | ||
|
|
||
| DDP's performance advantage comes from overlapping allreduce collectives with computations during backwards. | ||
| AotAutograd prevents this overlap when used with TorchDynamo for compiling a whole forward and whole backward graph, |
There was a problem hiding this comment.
I would imagine the DDP audience may not know what AotAutograd is, I'd rather expanding on this a bit more
Maybe a picture would help make things clearer
There was a problem hiding this comment.
Do you think it's better to duplicate the picture/explanation here, or would a link out to @davidberard98's blog suffice? He explains it well and has pictures
There was a problem hiding this comment.
ok. link is a paragraph below, but i could move it up if you think it would help.
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 4 additional jobs have failed, first few of them are: windows-binary-libtorch-debug ,windows-binary-libtorch-debug / libtorch-cpu-shared-with-deps-debug-test ,trunk ,trunk / win-vs2019-cuda11.6-py3 / test (force_on_cpu, 1, 1, windows.4xlarge) Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge -f "unrelated CI fail" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Pull Request resolved: pytorch#89096 Approved by: https://github.com/msaroufim
Stack from ghstack (oldest at bottom):