[FSDP] Doc to explain running submodules #86343

rohan-varma · 2022-10-06T00:16:01Z

Stack from ghstack (oldest at bottom):

[ghstack-poisoned]

pytorch-bot · 2022-10-06T00:16:07Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86343

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 2f96190:

The following jobs have failed:

linux-bionic-cuda11.7-py3.10-gcc7 / test (default, 1, 4, linux.4xlarge.nvidia.gpu)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 8677a06 Pull Request resolved: #86343

awgu

Thanks for adding this note!

awgu · 2022-10-06T00:39:37Z

torch/distributed/fsdp/fully_sharded_data_parallel.py


+    .. note:
+        Attempting to run the forward pass of a submodule that is contained in an
+        FSDP unit is not supported and will result in errors. This is because the


nit: I think maybe we should converge to just saying FSDP instance (where instance is the technical OOP term).

Suggested change

FSDP unit is not supported and will result in errors. This is because the

FSDP instance is not supported and will result in errors. This is because the

awgu · 2022-10-06T00:40:29Z

torch/distributed/fsdp/fully_sharded_data_parallel.py

+        Attempting to run the forward pass of a submodule that is contained in an
+        FSDP unit is not supported and will result in errors. This is because the
+        submodule's parameters will be sharded, but it itself is not an FSDP instance,
+        so its forward pass will not materialize the full parameters appropriately.


nit: just a suggestion

Suggested change

so its forward pass will not materialize the full parameters appropriately.

so its forward pass will not all-gather the full parameters appropriately.

awgu · 2022-10-06T00:40:48Z

torch/distributed/fsdp/fully_sharded_data_parallel.py

+        submodule's parameters will be sharded, but it itself is not an FSDP instance,
+        so its forward pass will not materialize the full parameters appropriately.
+        This could potentially happen when attempting to run only the encoder of a
+        encoder-decoder model, and the encoder is not wrapped in its own FSDP unit. To


Suggested change

encoder-decoder model, and the encoder is not wrapped in its own FSDP unit. To

encoder-decoder model, and the encoder is not wrapped in its own FSDP instance. To

awgu · 2022-10-06T00:41:02Z

torch/distributed/fsdp/fully_sharded_data_parallel.py

+        so its forward pass will not materialize the full parameters appropriately.
+        This could potentially happen when attempting to run only the encoder of a
+        encoder-decoder model, and the encoder is not wrapped in its own FSDP unit. To
+        resolve this, please wrap the submodule in its own FSDP unit.


Suggested change

resolve this, please wrap the submodule in its own FSDP unit.

resolve this, please wrap the submodule in its own FSDP instance.

[ghstack-poisoned]

ghstack-source-id: 3d14822 Pull Request resolved: #86343

rohan-varma · 2022-10-06T23:07:57Z

failure is unrelated and probably flaky -

TestCommonCUDA.test_noncontiguous_samples_nn_functional_conv_transpose2d_cuda_float32 _
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/test_ops.py", line 468, in test_noncontiguous_samples
    self.assertEqual(actual, expected)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2454, in assertEqual
    assert_equal(
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

rohan-varma · 2022-10-06T23:08:54Z

@pytorchbot merge -f "Test failure unrelated"

pytorchmergebot · 2022-10-06T23:10:12Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

github-actions · 2022-10-06T23:12:04Z

Hey @rohan-varma.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Pull Request resolved: #86343 Approved by: https://github.com/awgu Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/f0977c4658c6f8c10e3342cf9a0249d5d23a3505 Reviewed By: seemethere Differential Revision: D40167195 Pulled By: rohan-varma fbshipit-source-id: 04b71cfa79da0b50a12815ecdda99a139bf4723b

[FSDP] Doc to explain running submodules

2ed7425

[ghstack-poisoned]

rohan-varma requested review from H-Huang, awgu, kwen2501, mingzhe09088, mrshenli, pritamdamania87 and zhaojuanmao as code owners October 6, 2022 00:16

This was referenced Oct 6, 2022

CheckpointSequential support non-reentrant #86331

Closed

[FSDP] assert to runtime error #86336

Closed

pytorch-bot bot added the release notes: distributed (sharded) release notes category label Oct 6, 2022

rohan-varma mentioned this pull request Oct 6, 2022

[FSDP] Fix clip_grad_norm for CPU offload #86337

Closed

facebook-github-bot added the cla signed label Oct 6, 2022

facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 6, 2022

rohan-varma added a commit that referenced this pull request Oct 6, 2022

[FSDP] Doc to explain running submodules

44a74ae

ghstack-source-id: 8677a06 Pull Request resolved: #86343

rohan-varma mentioned this pull request Oct 6, 2022

[FSDP] caffe2 error in forward method when using fsdp #82461

Closed

awgu approved these changes Oct 6, 2022

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 6, 2022

Update on "[FSDP] Doc to explain running submodules"

45ab4a5

[ghstack-poisoned]

Update on "[FSDP] Doc to explain running submodules"

2f96190

[ghstack-poisoned]

rohan-varma added a commit that referenced this pull request Oct 6, 2022

[FSDP] Doc to explain running submodules

cf83bda

ghstack-source-id: 3d14822 Pull Request resolved: #86343

This was referenced Oct 6, 2022

[FSDP][BE][Docs] Expose helper classes #85495

Closed

[FSDP] Refactor comm hooks directory #85496

Closed

pytorchmergebot added the Merged label Oct 6, 2022

pytorchmergebot closed this in f0977c4 Oct 6, 2022

rohan-varma mentioned this pull request Oct 14, 2022

[RFC] Separate CPU offload activation to its own wrapper #85459

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FSDP] Doc to explain running submodules #86343

[FSDP] Doc to explain running submodules #86343

Uh oh!

rohan-varma commented Oct 6, 2022 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 6, 2022 •

edited

Loading

Uh oh!

awgu left a comment

Uh oh!

awgu Oct 6, 2022

Uh oh!

awgu Oct 6, 2022

Uh oh!

awgu Oct 6, 2022

Uh oh!

awgu Oct 6, 2022

Uh oh!

rohan-varma commented Oct 6, 2022

Uh oh!

rohan-varma commented Oct 6, 2022

Uh oh!

pytorchmergebot commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	FSDP unit is not supported and will result in errors. This is because the
	FSDP instance is not supported and will result in errors. This is because the

	so its forward pass will not materialize the full parameters appropriately.
	so its forward pass will not all-gather the full parameters appropriately.

	encoder-decoder model, and the encoder is not wrapped in its own FSDP unit. To
	encoder-decoder model, and the encoder is not wrapped in its own FSDP instance. To

	resolve this, please wrap the submodule in its own FSDP unit.
	resolve this, please wrap the submodule in its own FSDP instance.

[FSDP] Doc to explain running submodules #86343

[FSDP] Doc to explain running submodules #86343

Uh oh!

Conversation

rohan-varma commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86343

❌ 1 Failures

Uh oh!

awgu left a comment

Choose a reason for hiding this comment

Uh oh!

awgu Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

awgu Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

awgu Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

awgu Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

rohan-varma commented Oct 6, 2022

Uh oh!

rohan-varma commented Oct 6, 2022

Uh oh!

pytorchmergebot commented Oct 6, 2022

Merge started

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rohan-varma commented Oct 6, 2022 •

edited

Loading

pytorch-bot bot commented Oct 6, 2022 •

edited

Loading