Skip to content

Make ASGD & RProp differentiable #86258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
emcastillo wants to merge 1 commit into from
Closed

Make ASGD & RProp differentiable #86258

emcastillo wants to merge 1 commit into from

Conversation

@emcastillo
Copy link
Collaborator

@emcastillo emcastillo commented Oct 5, 2022

Blocked by #86183

@emcastillo emcastillo requested a review from albanD as a code owner October 5, 2022 04:39
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 5, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86258

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 5 Pending

As of commit 85c9b52:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Oct 5, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@emcastillo emcastillo added module: optimizer Related to torch.optim release notes: nn release notes category topic: improvements topic category labels Oct 5, 2022
albanD
albanD reviewed Oct 5, 2022
View reviewed changes
Copy link
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work!

torch/optim/adamax.py Outdated
Copy link
Collaborator

@albanD albanD Oct 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to update the passed in exp_inf for this to work properly no?
exp_inf.copy_(torch.amax(norm_buf, 0, keepdim=False))

Copy link
Collaborator Author

@emcastillo emcastillo Oct 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!

torch/optim/nadam.py Outdated
Copy link
Collaborator

@albanD albanD Oct 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could you append: "by updating the grad and exp_avg directly and not using the scalar "value" argument of addcdiv.

btw on GPU, it might be better to do this as the item() would cause a sync cc @ptrblck if someone on your end has time to check this out?

@albanD
Copy link
Collaborator

albanD commented Oct 5, 2022

(you might want to rebase on top of master for CI to run properly)

@emcastillo emcastillo force-pushed the differentiable-others branch from ec8759e to 1217410 Compare October 6, 2022 05:41
@emcastillo
Copy link
Collaborator Author

emcastillo commented Oct 6, 2022

I added the comments in each of the PR and removed the dependences

albanD
albanD approved these changes Oct 6, 2022
View reviewed changes
Copy link
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM !

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 6, 2022
@emcastillo
Copy link
Collaborator Author

emcastillo commented Oct 11, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 11, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 11, 2022

Merge failed

Reason: This PR is too stale; the last push date was more than 3 days ago. Please rebase and try again.

Details for Dev Infra team Raised by workflow job

@emcastillo emcastillo force-pushed the differentiable-others branch from 1217410 to 8679373 Compare October 12, 2022 08:27
@emcastillo emcastillo force-pushed the differentiable-others branch from 8679373 to 85c9b52 Compare October 13, 2022 00:27
@emcastillo
Copy link
Collaborator Author

emcastillo commented Oct 13, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 13, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@albanD albanD albanD approved these changes

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request cla signed Merged module: optimizer Related to torch.optim open source release notes: nn release notes category topic: improvements topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@emcastillo @albanD @pytorchmergebot @facebook-github-bot @pytorchbot