[functorch] add new ensembling api, demonstrate in example #88850

samdow · 2022-11-10T23:13:58Z

Stack from ghstack (oldest at bottom):

[ghstack-poisoned]

pytorch-bot · 2022-11-10T23:14:00Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88850

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit fd510c3:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

functorch/examples/ensembling/parallel_train.py

[ghstack-poisoned]

zou3519

Directionally this looks good to me and the code looks correct. We should bikeshed the name of the API before this goes in

torch/_functorch/make_functional.py

functorch/examples/ensembling/parallel_train.py

zou3519 · 2022-12-16T19:58:20Z

functorch/examples/ensembling/parallel_train.py

+        for key in grad_weights:
+            new_weights[key] = weights[key] - grad_weights[key] * lr


(no action required) Honestly should just look like the following:

def sgd_update(weight, grad_weight): return weight - grad_weight * lr new_weights = pytree.tree_map(sgd_update, weights, grad_weights)

but our tree_map doesn't have those semantics yet (JAX's does).

functorch/examples/ensembling/parallel_train.py

test/functorch/test_eager_transforms.py

torch/_functorch/make_functional.py

[ghstack-poisoned]

zou3519

I had two last two points, then this should be good to go!

My first reaction at stack_ensembled_state was some confusion, but I can be convinced
Are there tests for the error cases?

torch/_functorch/functional_call.py

[ghstack-poisoned]

zou3519 · 2022-12-28T19:01:53Z

Looks like there are some erroneous submodule changes

zou3519

LGTM

torch/_functorch/functional_call.py

[ghstack-poisoned]

samdow · 2022-12-29T16:36:39Z

@pytorchbot merge

pytorchmergebot · 2022-12-29T16:38:29Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2022-12-29T17:08:47Z

Merge failed

Reason: The following mandatory check(s) failed (Rule superuser):

pull

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

samdow · 2023-01-03T16:28:25Z

@pytorchmergebot rebase

pytorchmergebot · 2023-01-03T16:30:13Z

@pytorchbot successfully started a rebase job. Check the current status here

[ghstack-poisoned]

pytorchmergebot · 2023-01-03T16:30:31Z

Successfully rebased gh/samdow/46/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/88850)

ghstack-source-id: bf8db67 Pull Request resolved: #88850

samdow · 2023-01-04T00:29:40Z

@pytorchmergebot merge

pytorchmergebot · 2023-01-04T00:33:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

add new ensembling api, demonstrate in example

b5b65c2

[ghstack-poisoned]

samdow mentioned this pull request Nov 10, 2022

[functorch] rewrite examples that use make_functional to use functional_call #88851

Closed

samdow changed the title ~~add new ensembling api, demonstrate in example~~ [functorch] add new ensembling api, demonstrate in example Nov 10, 2022

samdow added topic: docs topic category release notes: torch.func release notes category for torch.vmap or torch.func.* APIs labels Nov 10, 2022

samdow requested a review from zou3519 November 15, 2022 17:29

samdow mentioned this pull request Nov 17, 2022

[functorch] add functorch functional_call, update tests to test this #89213

Closed

zou3519 reviewed Nov 17, 2022

View reviewed changes

functorch/examples/ensembling/parallel_train.py Outdated Show resolved Hide resolved

samdow added 3 commits December 15, 2022 15:19

Update on "[functorch] add new ensembling api, demonstrate in example"

d6986cd

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

630a7e3

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

23a3e1c

[ghstack-poisoned]

zou3519 reviewed Dec 16, 2022

View reviewed changes

samdow added 4 commits December 19, 2022 11:11

Update on "[functorch] add new ensembling api, demonstrate in example"

7bfe6f7

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

4969570

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

98379ab

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

336c4fd

[ghstack-poisoned]

samdow mentioned this pull request Dec 21, 2022

[functorch] excise remaining functorch imports from examples #91282

Closed

samdow added 3 commits December 22, 2022 13:02

Update on "[functorch] add new ensembling api, demonstrate in example"

d0e000e

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

45aa7c5

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

18fd960

[ghstack-poisoned]

zou3519 reviewed Dec 22, 2022

View reviewed changes

torch/_functorch/functional_call.py Outdated Show resolved Hide resolved

torch/_functorch/functional_call.py Outdated Show resolved Hide resolved

torch/_functorch/functional_call.py Show resolved Hide resolved

Update on "[functorch] add new ensembling api, demonstrate in example"

4f4f3da

[ghstack-poisoned]

samdow mentioned this pull request Dec 27, 2022

[functorch] move batch_norm_replacement to torch.func #91412

Closed

samdow added 2 commits December 27, 2022 15:38

Update on "[functorch] add new ensembling api, demonstrate in example"

c85d13b

[ghstack-poisoned]

Update on "[functorch] add new ensembling api, demonstrate in example"

9f826fd

[ghstack-poisoned]

zou3519 self-requested a review December 28, 2022 14:27

zou3519 approved these changes Dec 28, 2022

View reviewed changes

torch/_functorch/functional_call.py Outdated Show resolved Hide resolved

torch/_functorch/functional_call.py Show resolved Hide resolved

Update on "[functorch] add new ensembling api, demonstrate in example"

ebc7a3e

[ghstack-poisoned]

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 29, 2022

Update on "[functorch] add new ensembling api, demonstrate in example"

fd510c3

[ghstack-poisoned]

pytorchmergebot pushed a commit that referenced this pull request Jan 3, 2023

add new ensembling api, demonstrate in example

53189e4

ghstack-source-id: bf8db67 Pull Request resolved: #88850

pytorchmergebot added the Merged label Jan 4, 2023

pytorchmergebot closed this in 162474d Jan 4, 2023

facebook-github-bot deleted the gh/samdow/46/head branch June 8, 2023 18:43

		for key in grad_weights:
		new_weights[key] = weights[key] - grad_weights[key] * lr

[functorch] add new ensembling api, demonstrate in example #88850

[functorch] add new ensembling api, demonstrate in example #88850

Uh oh!

Conversation

samdow commented Nov 10, 2022 • edited by pytorchmergebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88850

✅ No Failures

Uh oh!

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zou3519 Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zou3519 commented Dec 28, 2022

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

samdow commented Dec 29, 2022

Uh oh!

pytorchmergebot commented Dec 29, 2022

Merge started

Uh oh!

pytorchmergebot commented Dec 29, 2022

Merge failed

Uh oh!

samdow commented Jan 3, 2023

Uh oh!

pytorchmergebot commented Jan 3, 2023

Uh oh!

pytorchmergebot commented Jan 3, 2023

Uh oh!

samdow commented Jan 4, 2023

Uh oh!

pytorchmergebot commented Jan 4, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

samdow commented Nov 10, 2022 •

edited by pytorchmergebot

Loading

pytorch-bot bot commented Nov 10, 2022 •

edited

Loading