[FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118

fegin · 2023-01-13T00:12:33Z

Stack from ghstack (oldest at bottom):

-> [FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118
[FSDP][optim_state_dict] Fix _is_named_optimizer when the state is empty #93303

Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load.

Differential Revision: D42488022

…dict_to_load public Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

pytorch-bot · 2023-01-13T00:12:35Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92118

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5dd0379:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…dict_to_load public Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) ghstack-source-id: 177584342 Pull Request resolved: #92118

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 177753434 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 177918763 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 177990998 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 178293084 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 178328384 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

rohan-varma

LGTM, thanks!

rohan-varma · 2023-01-23T05:18:44Z

torch/distributed/fsdp/api.py

+
+@dataclass
+class LocalOptimStateDictConfig(OptimStateDictConfig):
+    offload_to_cpu: bool = False


why is the default overriden to false for local, but not sharded?

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 178572798 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

fegin · 2023-01-30T22:23:49Z

@pytorchbot merge

pytorchmergebot · 2023-01-30T22:26:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-01-30T22:26:13Z

Merge failed

Reason: This PR is too stale; the last push date was more than 3 days ago. Please rebase and try again. You can rebase by leaving the following comment on this PR:
@pytorchbot rebase

Details for Dev Infra team

Raised by workflow job

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 178828472 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

fegin · 2023-01-31T06:40:57Z

@pytorchbot merge -f "The failing test is not related."

pytorchmergebot · 2023-01-31T06:44:34Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-01-31T06:44:38Z

Merge failed

Reason: Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x aebd6d62c8818a388682864d1d6ae979a7eedd5c returned non-zero exit code 1

Auto-merging test/distributed/fsdp/test_fsdp_optim_state.py
CONFLICT (content): Merge conflict in test/distributed/fsdp/test_fsdp_optim_state.py
Auto-merging torch/distributed/fsdp/fully_sharded_data_parallel.py
error: could not apply aebd6d62c8... [FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".

Details for Dev Infra team

Raised by workflow job

…ptim_state_dict_to_load public" Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/) [ghstack-poisoned]

…dict_to_load public Pull Request resolved: #92118 Make optim_state_dict and optim_state_dict_to_load public APIs and consolidate them with state_dict by using the same state_dict_type to decide how to perform the optimizer state_dict save and load. ghstack-source-id: 178869730 Differential Revision: [D42488022](https://our.internmc.facebook.com/intern/diff/D42488022/)

fegin · 2023-02-02T08:02:21Z

@pytorchbot merge

pytorchmergebot · 2023-02-02T08:04:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

fegin requested review from H-Huang, awgu, kwen2501, mrshenli, rohan-varma, wanchaol and zhaojuanmao as code owners January 13, 2023 00:12

pytorch-bot bot added the release notes: distributed (fsdp) release notes category label Jan 13, 2023

fegin closed this Jan 13, 2023

fegin reopened this Jan 17, 2023

fegin added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 17, 2023

rohan-varma approved these changes Jan 25, 2023

View reviewed changes

pytorchmergebot added the Merged label Feb 2, 2023

pytorchmergebot closed this in 4b0f1cc Feb 2, 2023

facebook-github-bot deleted the gh/fegin/60/head branch June 8, 2023 17:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118

[FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118

Uh oh!

fegin commented Jan 13, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jan 13, 2023 •

edited

Loading

Uh oh!

rohan-varma left a comment

Uh oh!

rohan-varma Jan 23, 2023

Uh oh!

fegin commented Jan 30, 2023

Uh oh!

pytorchmergebot commented Jan 30, 2023

Uh oh!

pytorchmergebot commented Jan 30, 2023

Uh oh!

fegin commented Jan 31, 2023

Uh oh!

pytorchmergebot commented Jan 31, 2023

Uh oh!

pytorchmergebot commented Jan 31, 2023

Uh oh!

fegin commented Feb 2, 2023

Uh oh!

pytorchmergebot commented Feb 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118

[FSDP][optim_state_dict][10/N] Make optim_state_dict and optim_state_dict_to_load public #92118

Uh oh!

Conversation

fegin commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92118

✅ No Failures

Uh oh!

rohan-varma left a comment

Choose a reason for hiding this comment

Uh oh!

rohan-varma Jan 23, 2023

Choose a reason for hiding this comment

Uh oh!

fegin commented Jan 30, 2023

Uh oh!

pytorchmergebot commented Jan 30, 2023

Merge started

Uh oh!

pytorchmergebot commented Jan 30, 2023

Merge failed

Uh oh!

fegin commented Jan 31, 2023

Uh oh!

pytorchmergebot commented Jan 31, 2023

Merge started

Uh oh!

pytorchmergebot commented Jan 31, 2023

Merge failed

Uh oh!

fegin commented Feb 2, 2023

Uh oh!

pytorchmergebot commented Feb 2, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fegin commented Jan 13, 2023 •

edited

Loading

pytorch-bot bot commented Jan 13, 2023 •

edited

Loading