Skip to content

Add debugability comments to DDPOptimizer #89802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wconstab wants to merge 6 commits into from
Closed

Add debugability comments to DDPOptimizer #89802

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6dbce3a
Add debugability comments to DDPOptimizer
wconstab Nov 28, 2022
fbf3114
Update on "Add debugability comments to DDPOptimizer"
wconstab Nov 29, 2022
a3383be
Update on "Add debugability comments to DDPOptimizer"
wconstab Nov 29, 2022
8851e4f
Update on "Add debugability comments to DDPOptimizer"
wconstab Nov 29, 2022
72339d4
Update on "Add debugability comments to DDPOptimizer"
wconstab Jan 20, 2023
f552f60
Update on "Add debugability comments to DDPOptimizer"
wconstab Jan 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions torch/_dynamo/optimizations/distributed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,13 @@ class DDPOptimizer:
and therefore aren't allreduced by DDP. (They are broadcast during forward, but this is not covered by
DDPOptimizer)

Debugging
- Generally, it is easiest to debug DDPOptimizer in a single process program, using pdb.
- In many cases, the log messages are helpful (they show bucket size assignments)-
just configure torch._dynamo.config.log_level to info or debug.
- See `benchmarks/dynamo/distributed.py` for a simple harness that will run a toy model or a torchbench model
in a single process (or with torchrun, in multiple processes)

Args:
bucket_bytes_cap (int): Controls the size of buckets, in bytes, used to determine graphbreaks. Should be
set to match the equivalent parameter on the original DDP module.
Expand Down