[quant][pyper] Add embedding_bag weight quantize and dequantize ops #41293

supriyar · 2020-07-11T00:29:11Z

Stack from ghstack:

[quant][pyper] Add embedding_bag weight quantize and dequantize ops #41293 [quant][pyper] Add embedding_bag weight quantize and dequantize ops

Summary:

Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators.
This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module

Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point.

Test Plan:
python test/test_quantization.py TestQuantizedEmbeddingBag

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D22506700

Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: cfbca83 Pull Request resolved: #41293

dr-ci · 2020-07-11T02:49:33Z

💊 CI failures summary and remediations

As of commit d7b399d (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-CircleCI failure(s)

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-xenial-rocm3.5.1-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 13 times.

…ntize ops" Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 782b7da Pull Request resolved: #41293

radkris-git · 2020-07-13T19:28:57Z

aten/src/ATen/native/quantized/cpu/qembeddingbag_prepack.cpp

+  constexpr int NUM_ELEM_PER_BYTE = 8 / BIT_RATE;
+  TORCH_CHECK(
+      weight_contig.size(weight.dim() - 1) % NUM_ELEM_PER_BYTE == 0,
+      "FloatToFused4BitRowwiseQuantizedOp only works for the number of "


I think this is a nit.

You mean this check isn't required?

The error message says "FloatToFused4BitRowwiseQuantizedOp". :) It should be "qembeddingbag_4bit_prepack only works for the number of columns a multiple of 2".

test/quantization/test_quantized_op.py

…ntize ops" Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22506700](https://our.internmc.facebook.com/intern/diff/D22506700) [ghstack-poisoned]

vkuzo

lg, feel free to ignore the comments if this implementation will be replaced in the near future

vkuzo · 2020-07-13T22:27:32Z

aten/src/ATen/native/quantized/cpu/qembeddingbag_prepack.cpp

+  auto* output_data = output.data_ptr<uint8_t>();
+  const auto output_columns = output.size(output.dim() - 1);
+
+  for (int row = 0; row < embedding_rows; ++row) {


does performance matter, or is this a reference implementation? Could probably parallelize if needed (same for the other op)

This op is a temporary solution until we de-couple quantize and packing. We can re-visit optimizations then.

makes sense

aten/src/ATen/native/quantized/cpu/qembeddingbag_prepack.cpp

vkuzo · 2020-07-13T22:31:13Z

aten/src/ATen/native/quantized/cpu/qembeddingbag_prepack.cpp

+    const float* input_row = weight_data + row * embedding_cols;
+    std::uint8_t* output_row = output_data + row * output_columns;
+
+    at::Half* output_row_scale_zp = reinterpret_cast<at::Half*>(


optional readability nit: if this is packed at the end of a row, maybe we can move the code down to be below the weight packing, so the code structure follows the data format?

aten/src/ATen/native/quantized/cpu/qembeddingbag_prepack.cpp

aten/src/ATen/native/quantized/cpu/qembeddingbag_unpack.cpp

…ntize ops" Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22506700](https://our.internmc.facebook.com/intern/diff/D22506700) [ghstack-poisoned]

Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 80fec1a Pull Request resolved: #41293

…ntize ops" Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22506700](https://our.internmc.facebook.com/intern/diff/D22506700) [ghstack-poisoned]

Summary: Add new operators that does quantize and packing for 8 bit and 4 bit embedding bag operators. This is an initial change to help unblock testing. This will be follwed by adding graph mode passes to enable quantization of embedding_bag module Note to reviewers: Future PRs will replace this op with a separate quantize and pack operator and add support for floating point scale and zero point. Test Plan: python test/test_quantization.py TestQuantizedEmbeddingBag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: fe24a52 Pull Request resolved: #41293

facebook-github-bot · 2020-07-16T00:15:14Z

This pull request has been merged in 008ab27.

facebook-github-bot · 2020-07-16T00:15:25Z

This pull request has been merged in 008ab27.

supriyar requested review from qizzzh, radkris-git, raghuramank100 and vkuzo July 13, 2020 17:46

radkris-git reviewed Jul 13, 2020

View reviewed changes

vkuzo reviewed Jul 13, 2020

View reviewed changes

vkuzo approved these changes Jul 14, 2020

View reviewed changes

radkris-git approved these changes Jul 14, 2020

View reviewed changes

facebook-github-bot closed this in 008ab27 Jul 15, 2020

facebook-github-bot added the merged label Jul 16, 2020

supriyar mentioned this pull request Jul 17, 2020

[quant] Add Graph Mode Passes to quantize EmbeddingBag operators #41612

Closed

facebook-github-bot deleted the gh/supriyar/149/head branch July 19, 2020 14:18

mruberry added the Merged label Oct 28, 2020

[quant][pyper] Add embedding_bag weight quantize and dequantize ops #41293

[quant][pyper] Add embedding_bag weight quantize and dequantize ops #41293

Uh oh!

Conversation

supriyar commented Jul 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Jul 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

ci.pytorch.org: 1 failed

Uh oh!

radkris-git Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

supriyar Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

radkris-git Jul 14, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vkuzo left a comment

Choose a reason for hiding this comment

Uh oh!

vkuzo Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

supriyar Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

vkuzo Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vkuzo Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

facebook-github-bot commented Jul 16, 2020

Uh oh!

facebook-github-bot commented Jul 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

supriyar commented Jul 11, 2020 •

edited

Loading

dr-ci bot commented Jul 11, 2020 •

edited

Loading