Make DDPOptimizer work with torch._dynamo.explain() #94749
Conversation
GraphModules that were created during DDPOptimizer graph breaking lacked `compile_subgraph_reason`, which caused an exception when running .explain(). Now the reason is provided and users can use .explain() to find out that DDPOptimizer is causing graph breaks. Fixes #94579 [ghstack-poisoned]
wconstab
requested review from
H-Huang,
awgu,
fegin,
kwen2501,
mrshenli,
rohan-varma,
wanchaol and
zhaojuanmao
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94749
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 FailuresAs of commit f8025ad: NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| _, _, _, _, break_reasons, _ = torch._dynamo.explain(ddp_m, inputs) | ||
| self.assertEqual(len(break_reasons), 3) |
There was a problem hiding this comment.
nit, this might be a bit saner:
| _, _, _, _, break_reasons, _ = torch._dynamo.explain(ddp_m, inputs) | |
| self.assertEqual(len(break_reasons), 3) | |
| explain_out = torch._dynamo.explain(ddp_m, inputs) | |
| break_reasons = explain_out[4] | |
| self.assertEqual(len(break_reasons), 3) |
| input_mod.recompile() | ||
| input_mod.compile_subgraph_reason = GraphCompileReason( | ||
| "DDPOptimizer intentional graph-break to improve communication overlap." | ||
| " Set `torch._dynamo.config.optimize_ddp = False` to disable", |
There was a problem hiding this comment.
is there a longer a note or comment on this flag?
There was a problem hiding this comment.
yea just added a pointer. good idea.
GraphModules that were created during DDPOptimizer graph breaking lacked `compile_subgraph_reason`, which caused an exception when running .explain(). Now the reason is provided and users can use .explain() to find out that DDPOptimizer is causing graph breaks. Fixes #94579 cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / test (default, 2, 2, macos-m1-12) Details for Dev Infra teamRaised by workflow job |
GraphModules that were created during DDPOptimizer graph breaking lacked `compile_subgraph_reason`, which caused an exception when running .explain(). Now the reason is provided and users can use .explain() to find out that DDPOptimizer is causing graph breaks. Fixes #94579 cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]
GraphModules that were created during DDPOptimizer graph breaking lacked `compile_subgraph_reason`, which caused an exception when running .explain(). Now the reason is provided and users can use .explain() to find out that DDPOptimizer is causing graph breaks. Fixes #94579 ghstack-source-id: fa51cd1 Pull Request resolved: #94749
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: linux-binary-manywheel / manywheel-py3_8-cuda11_7-with-pypi-cudnn-test / test Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge -f "Broken trunk" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
GraphModules that were created during DDPOptimizer graph breaking
lacked
compile_subgraph_reason, which caused an exception whenrunning .explain().
Now the reason is provided and users can use .explain() to find out
that DDPOptimizer is causing graph breaks.
Fixes #94579
cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire