Skip to content

Optimize SiLU (Swish) op in PyTorch #42976

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
xiaomengy wants to merge 1 commit into from
Closed

Optimize SiLU (Swish) op in PyTorch #42976

xiaomengy wants to merge 1 commit into from

Conversation

@xiaomengy
Copy link
Contributor

@xiaomengy xiaomengy commented Aug 13, 2020 •
edited
Loading

Summary:
Optimize SiLU (Swish) op in PyTorch.

Some benchmark result

input = torch.rand(1024, 32768, dtype=torch.float, device="cpu")
forward: 221ms -> 133ms
backward: 600ms -> 170ms

input = torch.rand(1024, 32768, dtype=torch.double, device="cpu")
forward: 479ms -> 297ms
backward: 1438ms -> 387ms

input = torch.rand(8192, 32768, dtype=torch.float, device="cuda")
forward: 24.34ms -> 9.83ms
backward: 97.05ms -> 29.03ms

input = torch.rand(4096, 32768, dtype=torch.double, device="cuda")
forward: 44.24ms -> 30.15ms
backward: 126.21ms -> 49.68ms

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "SiLU"

Differential Revision: D23093593

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 13, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@xiaomengy xiaomengy requested a review from ngimel August 13, 2020 05:00
@dr-ci
Copy link

dr-ci bot commented Aug 13, 2020 •
edited
Loading

💊 CI failures summary and remediations

As of commit ac3c803 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 40 times.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 13, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 13, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@xiaomengy
Copy link
Contributor Author

xiaomengy commented Aug 13, 2020

linke #43017

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 13, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

ngimel
ngimel reviewed Aug 13, 2020
View reviewed changes
gchanan
gchanan reviewed Aug 14, 2020
View reviewed changes
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 14, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 14, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

ngimel
ngimel reviewed Aug 14, 2020
View reviewed changes
Copy link
Collaborator

@ngimel ngimel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thank you!

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 14, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

ngimel
ngimel reviewed Aug 14, 2020
View reviewed changes
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 14, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@xiaomengy @facebook-github-bot
Optimize SiLU (Swish) op in PyTorch (pytorch#42976)
ac3c803 
Summary:
Pull Request resolved: pytorch#42976

Optimize SiLU (Swish) op in PyTorch.

Some benchmark result

input = torch.rand(1024, 32768, dtype=torch.float, device="cpu")
forward: 221ms -> 133ms
backward: 600ms -> 170ms

input = torch.rand(1024, 32768, dtype=torch.double, device="cpu")
forward: 479ms -> 297ms
backward: 1438ms -> 387ms

input = torch.rand(8192, 32768, dtype=torch.float, device="cuda")
forward: 24.34ms -> 9.83ms
backward: 97.05ms -> 29.03ms

input = torch.rand(4096, 32768, dtype=torch.double, device="cuda")
forward: 44.24ms -> 30.15ms
backward: 126.21ms -> 49.68ms

Test Plan: buck test mode/dev-nosan //caffe2/test:nn -- "SiLU"

Reviewed By: houseroad

Differential Revision: D23093593

fbshipit-source-id: ca6129d9fce76b569ac7ea0690dd38eac23c221d
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 15, 2020

This pull request was exported from Phabricator. Differential Revision: D23093593

@xiaomengy xiaomengy deleted the export-D23093593 branch August 16, 2020 20:39
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 16, 2020

This pull request has been merged in 4ae832e.

@glaringlee glaringlee mentioned this pull request Aug 17, 2020
Closed
@TFUsers TFUsers mentioned this pull request Oct 29, 2020
Closed
@hendrycks hendrycks mentioned this pull request Jan 7, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gchanan gchanan gchanan left review comments

@ngimel ngimel ngimel approved these changes

Assignees

No one assigned

Labels

fb-exported Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xiaomengy @facebook-github-bot @gchanan @ngimel @mruberry