[quant] Add embeddingbag_prepack function that works on quantized tensor. #42762
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sor. Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
This was referenced Aug 7, 2020
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
💊 CI failures summary and remediationsAs of commit bc1d0f9 (more details on the Dr. CI page):
ci.pytorch.org: 1 failedThis comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 53 times. |
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
supriyar
added a commit
that referenced
this pull request
Aug 7, 2020
…sor. Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: b26323b Pull Request resolved: #42762
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
supriyar
added a commit
that referenced
this pull request
Aug 10, 2020
…sor. Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 05ffb75 Pull Request resolved: #42762
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
jerryzh168
reviewed
Aug 11, 2020
| // TODO: Extend this to support 4-bits once 4-bit qtensor support is added. | ||
| Tensor qembeddingbag_prepack(at::Tensor qweight) { | ||
| Tensor weight_contig = qweight.contiguous(qweight.suggest_memory_format()); | ||
| const uint8_t* weight_data = |
Contributor
There was a problem hiding this comment.
do we want to check the dtype here?
Contributor
Author
There was a problem hiding this comment.
Sure, I can add a check. We don't check dtype in many ops though, probably because data_ptr would error out for incorrect type
jerryzh168
reviewed
Aug 11, 2020
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23070632](https://our.internmc.facebook.com/intern/diff/D23070632) [ghstack-poisoned]
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23070632](https://our.internmc.facebook.com/intern/diff/D23070632) [ghstack-poisoned]
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23070632](https://our.internmc.facebook.com/intern/diff/D23070632) [ghstack-poisoned]
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23070632](https://our.internmc.facebook.com/intern/diff/D23070632) [ghstack-poisoned]
…antized tensor." Summary: Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data. This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too. Note -In the following change I will add TorchBind support for this to support serialization of packed weights. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23070632](https://our.internmc.facebook.com/intern/diff/D23070632) [ghstack-poisoned]
Contributor
|
This pull request has been merged in 7632a9b. |
Contributor
|
This pull request has been merged in 7632a9b. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary:
Use a prepack function that accepts qtensor as an input. The output is a byte tensor with packed data.
This is currently implemented only for 8-bit. In the future once we add 4-bit support this function will be extended to support that too.
Note -In the following change I will add TorchBind support for this to support serialization of packed weights.
Test Plan:
python test/test_quantization.py TestQuantizedEmbeddingBag
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: D23070632