Skip to content

Use cub 1.15's latest scan-by-key algorithm to replace thrust for Embedding.cu and EmbeddingBag.cu #66580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zasdfgbnm wants to merge 51 commits into from
Closed

Use cub 1.15's latest scan-by-key algorithm to replace thrust for Embedding.cu and EmbeddingBag.cu #66580

zasdfgbnm wants to merge 51 commits into from

Conversation

@zasdfgbnm
Copy link
Collaborator

@zasdfgbnm zasdfgbnm commented Oct 13, 2021 •
edited
Loading

No description provided.

@zasdfgbnm zasdfgbnm mentioned this pull request Jan 6, 2022
Closed
@ngimel
Copy link
Collaborator

ngimel commented Feb 9, 2022

Ok, I trust it that your internal 11.6 tests are passing?

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Feb 9, 2022

@ngimel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@zasdfgbnm
Copy link
Collaborator Author

zasdfgbnm commented Feb 10, 2022

I run tests locally many times, both with CUDA 11.6 and with nightly cub. After merging, our nightly CI will start testing this daily with CUDA 11.6.

facebook-github-bot pushed a commit that referenced this pull request Feb 10, 2022
@zasdfgbnm @facebook-github-bot
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
51cff8c 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: #66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2022

Hey @zasdfgbnm.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

@zasdfgbnm zasdfgbnm deleted the cub-1.15 branch February 10, 2022 04:35
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 15, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
2b263b1 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 15, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
d850ce2 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 15, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
fcea161 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 16, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
4264517 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 16, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
cdaa2ba 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
0787910 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
08ac977 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
e5ba1ee 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
a18aaca 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
16b21ad 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
0bd78a8 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 17, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
072573f 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 20, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
944d2f4 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 20, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
8011733 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 20, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
8a98805 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Feb 21, 2022
@zasdfgbnm @cyyever
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
4d87d8f 
…edding.cu and EmbeddingBag.cu (#66580)

Summary: Pull Request resolved: pytorch/pytorch#66580

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8cb1de725bca52d5137b01b16d054b95f63)
PaliC added a commit that referenced this pull request Mar 1, 2022
@PaliC
Use cub 1.15's latest scan-by-key algorithm to replace thrust for Emb…
045b10a 
…edding.cu and EmbeddingBag.cu (#66580)

Summary:

Reviewed By: mruberry

Differential Revision: D34116388

Pulled By: ngimel

fbshipit-source-id: 2e8936ca7c10f96a8e7a5696248f56bf87290d6e
(cherry picked from commit 51cff8c)

[ghstack-poisoned]
facebook-github-bot pushed a commit that referenced this pull request Mar 16, 2022
@zasdfgbnm @facebook-github-bot
Remove sync in embedding (#70943)
d959fa4 
Summary:
This together with #66580 and #68376 will remove all syncs in embedding.

This PR includes #68376, please review after merging #68376

This PR introduces perf regressions and increases memory usage:
- `exclusive_sum` is now computing the entire `numel` elements instead of `num_of_segments` elements, and the trailing `numel - num_of_segments` results will be discarded.
- Some memory allocation now needs `numel` spaces instead of `num_of_segments` or `num_of_partial_segments`.

These are the prices we must pay in order to get a sync-free implementation.

I haven't done any benchmark yet. I will do it later.

Pull Request resolved: #70943

Reviewed By: H-Huang

Differential Revision: D34881660

Pulled By: ngimel

fbshipit-source-id: b0760fa33608c46cd4145ceb09878bf94a9f959d
pytorchmergebot pushed a commit that referenced this pull request Mar 16, 2022
@zasdfgbnm @pytorchmergebot
Remove sync in embedding (#70943)
11dc158 
Summary:
This together with #66580 and #68376 will remove all syncs in embedding.

This PR includes #68376, please review after merging #68376

This PR introduces perf regressions and increases memory usage:
- `exclusive_sum` is now computing the entire `numel` elements instead of `num_of_segments` elements, and the trailing `numel - num_of_segments` results will be discarded.
- Some memory allocation now needs `numel` spaces instead of `num_of_segments` or `num_of_partial_segments`.

These are the prices we must pay in order to get a sync-free implementation.

I haven't done any benchmark yet. I will do it later.

Pull Request resolved: #70943

Reviewed By: H-Huang

Differential Revision: D34881660

Pulled By: ngimel

fbshipit-source-id: b0760fa33608c46cd4145ceb09878bf94a9f959d
(cherry picked from commit d959fa4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngimel ngimel ngimel approved these changes

Assignees

No one assigned

Labels

cla signed module: nn Related to torch.nn oncall: jit Add this issue/PR to JIT oncall triage queue open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zasdfgbnm @facebook-github-bot @ngimel @pytorchbot @jbschlosser