3575 bug shfl down sync bug causes undefined behavior #3576
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edwinsolisf
reviewed
Jan 4, 2025
edwinsolisf
previously approved these changes
Jan 4, 2025
Contributor
edwinsolisf
left a comment
edwinsolisf
left a comment
There was a problem hiding this comment.
Tested on Windows and Ubuntu-22.04, passes all tests
…rp primitives and calls the new primitives for CUDA versions greater than 9 and the old ones for older CUDA versions. The new primitives have an additional argument which is a mask of the warp threads that are participating in the operation. The old primitives always involve all the threads in a warp. The wrapper routines originally allowed you to specify the mask which was ignored for the old primitives but this has now been removed. This is because if an old version of CUDA is being used then all threads must enter the wrapper routine and if a new version of CUDA is being used only the threads corresponding to the mask must enter. If threads outside the mask enter the routine then the behavior is undefined. In CUDA versions <=12.2 the primitive executes without any errors given however in later versions of CUDA a warp illegal instruction exception will be thrown. In order to preserve the same behavior of these wrapper functions for old and new versions of CUDA, the mask is always set to all threads in a warp for the new primitives. The specific new primitive can always be called with a custom mask which is already done elsewhere in the reduce_by_key routine.
christophe-murphy
force-pushed
the
3575-bug-__shfl_down_sync-bug-causes-undefined-behavior
branch
from
a687394 to
8355779
Compare
edwinsolisf
approved these changes
Jan 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix for bug where warp primitives were called by threads outside mask, triggering a warp illegal instruction exception in CUDA versions > 12.2
Description
A bug has been uncovered by CUDA versions newer than 12.2 where calls to the __warp_down_sync primitive are being made by threads outside of the thread mask in the reduce_by_key header. Although this results in undefined behavior, in prior CUDA versions the primitive would execute successfully. This has changed since in newer versions since 12.2 where a warp illegal instruction exception is thrown.
The primitive wrapper functions header has been modified so that the full mask is always used to ensure the same behavior with old and new versions of CUDA. If a different mask is required, then the specific new version of the primitive can be called.
Fixes: #3575
Checklist