[RELAND] [cuDNN] Add a new optimized cuDNN RNN algorithm for small RNN hidden_size #73211
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
February 22, 2022 08:30
…MALL_H algorithm (pytorch#72089)" This reverts commit c93d6f9.
CI Flow Status⚛️ CI FlowRuleset - Version:
|
Contributor
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit eeb5945 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Please report bugs/suggestions to the (internal) Dr. CI Users group. |
Collaborator
|
What about regular persistent algorithm? Should it bail out when inputs are packed? Does it do it (if so, where? It's not in the algo selection function)? |
Collaborator
|
|
Collaborator
Author
|
The other persistent algo already bails out if the input is packed (and it only triggers for fp16 data). |
Collaborator
Author
|
I reworked the checks, hopefully good now. |
Collaborator
|
Does PERSIST_STATIC_SMALL_H support arbitrary hidden and input sizes? For regular persist algorithm there are checks that those are multiples of 128 or 64 or something. |
Collaborator
Author
No idea. @xwang233 did you discuss this at all with cudnn people during the original PR? |
Collaborator
|
Only small hidden sizes are supported and those are already in the PR code. All input sizes should be supported. |
saketh-are
added
the
triaged
ngimel
approved these changes
Mar 7, 2022
Contributor
|
@ngimel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
facebook-github-bot
pushed a commit
that referenced
this pull request
Mar 7, 2022
…N hidden_size (#73211) Summary: #62143 was reverted (#72089) because, when running native tests internally with cudnn and GPUs such that `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` was used, we hit some `CUDNN_STATUS_NOT_SUPPORTED` errors. Based on https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#features-of-rnn-functions and experiments, I strongly suspect the errors were because `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` doesn't support variable sequence lengths in the batch. This PR restores #62143 and adds a bailout condition if the input is a packed batch that might have different sequence lengths per element. Question for review: Do we also need to add a bailout condition if the input is double precision? Pull Request resolved: #73211 Reviewed By: ejguan Differential Revision: D34688016 Pulled By: ngimel fbshipit-source-id: e7335c4701dabc7d0b36ebdb6414c4353a71ee91
Contributor
|
Hey @mcarilli. |
cyyever
pushed a commit
to cyyever/pytorch_private
that referenced
this pull request
Mar 9, 2022
…N hidden_size (#73211) Summary: pytorch/pytorch#62143 was reverted (pytorch/pytorch#72089) because, when running native tests internally with cudnn and GPUs such that `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` was used, we hit some `CUDNN_STATUS_NOT_SUPPORTED` errors. Based on https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#features-of-rnn-functions and experiments, I strongly suspect the errors were because `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` doesn't support variable sequence lengths in the batch. This PR restores pytorch/pytorch#62143 and adds a bailout condition if the input is a packed batch that might have different sequence lengths per element. Question for review: Do we also need to add a bailout condition if the input is double precision? Pull Request resolved: pytorch/pytorch#73211 Reviewed By: ejguan Differential Revision: D34688016 Pulled By: ngimel fbshipit-source-id: e7335c4701dabc7d0b36ebdb6414c4353a71ee91 (cherry picked from commit b9023bfd1c31eb9a38bf0552a20412e9a4e60b91)
cyyever
pushed a commit
to cyyever/pytorch_private
that referenced
this pull request
Mar 9, 2022
…N hidden_size (#73211) Summary: pytorch/pytorch#62143 was reverted (pytorch/pytorch#72089) because, when running native tests internally with cudnn and GPUs such that `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` was used, we hit some `CUDNN_STATUS_NOT_SUPPORTED` errors. Based on https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#features-of-rnn-functions and experiments, I strongly suspect the errors were because `CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H` doesn't support variable sequence lengths in the batch. This PR restores pytorch/pytorch#62143 and adds a bailout condition if the input is a packed batch that might have different sequence lengths per element. Question for review: Do we also need to add a bailout condition if the input is double precision? Pull Request resolved: pytorch/pytorch#73211 Reviewed By: ejguan Differential Revision: D34688016 Pulled By: ngimel fbshipit-source-id: e7335c4701dabc7d0b36ebdb6414c4353a71ee91 (cherry picked from commit b9023bfd1c31eb9a38bf0552a20412e9a4e60b91)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#62143 was reverted (#72089) because, when running native tests internally with cudnn and GPUs such that
CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_Hwas used, we hit someCUDNN_STATUS_NOT_SUPPORTEDerrors.Based on https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#features-of-rnn-functions and experiments, I strongly suspect the errors were because
CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_Hdoesn't support variable sequence lengths in the batch.This PR restores #62143 and adds a bailout condition if the input is a packed batch that might have different sequence lengths per element.
Question for review: Do we also need to add a bailout condition if the input is double precision?