Speed up CPU threshold and relu implementation #13182
Conversation
colesbury
commented
Oct 26, 2018
colesbury
commented
The previous threshold implementation was not vectorized or parallelized. This speeds up ResNet-50 CPU inference [1] from ~88 ms to ~67 ms CPU timings: https://gist.github.com/colesbury/d0d1be6974841d62696dbde329a8fde8 1 thread (before vs. after) 10240: 17.4 us vs. 6.9 µs per loop 102400: 141 us vs. 39.8 µs per loop 16 threads (before vs. after) 10240: 17.4 us vs. 6.7 µs per loop 102400: 141 us vs. 14.3 µs per loop CUDA timings are not measurably different. [1]: compiled with MKL-DNN, 8 threads, batch norm merged into convolutions https://gist.github.com/colesbury/8a64897dae97558b3b82da665048c782
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
colesbury has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
We have some internal code that calls relu() on LongTensors. I'm not sure we deliberately intended to support that, but it's easy enough to make it keep working.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
colesbury has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
| variants: function | ||
|
|
||
| - func: threshold_(Tensor self, Scalar threshold, Scalar value) -> Tensor | ||
| variants: function |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| variants: function | ||
|
|
||
| - func: threshold_out(Tensor result, Tensor self, Scalar threshold, Scalar value) -> Tensor | ||
| variants: function |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| self: threshold_backward(grad, self, 0, 0) | ||
| self: threshold_backward(grad, self, 0) | ||
|
|
||
| - name: relu_(Tensor self) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
|
||
| - name: threshold_forward(Tensor self, Scalar threshold, Scalar value) | ||
| self: threshold_backward(grad, self, threshold, value) | ||
| - name: threshold(Tensor self, Scalar threshold, Scalar value) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@colesbury has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@colesbury has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@colesbury is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: ``` The previous threshold implementation was not vectorized or parallelized. This speeds up ResNet-50 CPU inference [1] from ~88 ms to ~67 ms CPU timings: https://gist.github.com/colesbury/d0d1be6974841d62696dbde329a8fde8 1 thread (before vs. after) 10240: 17.4 us vs. 6.9 µs per loop 102400: 141 us vs. 39.8 µs per loop 16 threads (before vs. after) 10240: 17.4 us vs. 6.7 µs per loop 102400: 141 us vs. 14.3 µs per loop CUDA timings are not measurably different. [1]: compiled with MKL-DNN, 8 threads, batch norm merged into convolutions https://gist.github.com/colesbury/8a64897dae97558b3b82da665048c782 ``` Pull Request resolved: pytorch/pytorch#13182 Reviewed By: soumith Differential Revision: D12825105 Pulled By: colesbury fbshipit-source-id: 557da608ebb87db8a04adbb0d2882af4f2eb3c15
4 participants
