[quant][graphmode] Support quantization for `aten::apend` #40743

jerryzh168 · 2020-06-29T23:03:51Z

Stack from ghstack:

[quant][graphmode][fix] Print the node in error message #40889 [quant][graphmode][fix] Print the node in error message
[quant][graphmode][fix] Fold conv bn #40865 [quant][graphmode][fix] Fold conv bn
[quant][graphmode] Support quantization for aten::apend #40743 [quant][graphmode] Support quantization for aten::apend

Summary:
aten::append modifies input inplace and the output is ignored, these ops are not
supported right now, so we'll need to first make aten::append non-inplace
by change

ignored = aten::append(list, x)

to

x_list = aten::ListConstruct(x)
result = aten::add(list, x_list)

and then quantize the aten::add instead.

Test Plan:

TestQuantizeJitOps.test_general_shape_ops

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D22302151

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

dr-ci · 2020-06-29T23:07:00Z

💊 CI failures summary and remediations

As of commit 86b26ec (more details on the Dr. CI page):

1/1 failures introduced in this PR

1 job timed out:

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_test

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 28 times.

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: bef4f20 Pull Request resolved: #40743

jerryzh168 · 2020-06-29T23:25:52Z

This changes the underlying behavior: We are consuming more memory than before, is there a way to support append directly? How is the behavior of append different from that of cat?

aten::append is inplace, and the output is ignored. If script/trace can generate a non inplace append then we'll be able to quantize append directly, similar to concat

raghuramank100 · 2020-06-29T23:27:21Z

torch/csrc/jit/passes/quantization/insert_observers.cpp

 }

+void makeAppendNonInplace(std::shared_ptr<Graph>& graph) {
+  std::string append_pattern = R"IR(


Can you elaborate more on why directly supporting append is not possible?

All current ops including inplace ops assumes that the output will be consumed by the following ops. To break this assumption, we'll need to introduce substantial changes/hacks, I don't think it's worth the effort

btw, if there are perf problems we can easily add a pass to change the add to append in the end

raghuramank100 · 2020-06-29T23:28:47Z

torch/csrc/jit/passes/quantization/helper.cpp

-  } else if (n->kind() == Symbol::aten("append")) {
-    // notice that append is an op that changes input inplace
-    return {n->input(0), n->input(1)};
+  } else if (isListAdd(n)) {


What is the rule for list add? Do we assume that all the tensors in the list have a dequantize op prior to the add?

this is the transformed "append", as shown in the description. we will check if the inputs are produced with dequantize to make sure the inputs are quantized

the rule of list add:

%y = aten::add(%list, %x)

we'll check if %list is empty list, if it is, then the pass through list for %y is {%x}, otherwise, it is {%list, %x}

How do you check if the existing list has only dequantized tensors?. I see how you can do it for the input %x, dont follow how the check is done for %list

There are two cases for %list:
1.
one case is same as %x

%list = aten::dequantize(%list_quant) %result = aten::add(%list, %x_list)

we check if both %list and %x_list is produced by dequantize or not
2.
another case is when list is empty, it can be considered as containing quantized tensors

%list : Tensor[] = prim::ListConstruct() %result = aten::add(%list, %x_list)

in this case we only need to check if %x_list is produced by dequantize.

My question is on how we check if %list is produced by dequantize or not. Do we iterate over all elements in the list?

no we don't, we just check if %list is produced by dequantize or not

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: c5f2c6c Pull Request resolved: #40743

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: f79bc09 Pull Request resolved: #40743

jerryzh168 · 2020-06-30T01:30:36Z

@raghuramank100 I added a fix to change add back to append in finalize, please take a look again.

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22302151](https://our.internmc.facebook.com/intern/diff/D22302151) [ghstack-poisoned]

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: f5f959a Pull Request resolved: #40743

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22302151](https://our.internmc.facebook.com/intern/diff/D22302151) [ghstack-poisoned]

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: f7ceb92 Pull Request resolved: #40743

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22302151](https://our.internmc.facebook.com/intern/diff/D22302151) [ghstack-poisoned]

vkuzo · 2020-07-02T03:17:12Z

btw, where is the source code for aten::append? didn't find it in native_functions.yaml nor in the pytorch c++ docs (might have been looking in the wrong place)

vkuzo

accepting to unblock

jerryzh168 · 2020-07-02T18:39:05Z

btw, where is the source code for aten::append? didn't find it in native_functions.yaml nor in the pytorch c++ docs (might have been looking in the wrong place)

not sure it's probably something just in JIT

Summary: `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22302151](https://our.internmc.facebook.com/intern/diff/D22302151) [ghstack-poisoned]

jerryzh168 · 2020-07-02T18:55:57Z

btw, where is the source code for aten::append? didn't find it in native_functions.yaml nor in the pytorch c++ docs (might have been looking in the wrong place)

found it: https://codebrowser.bddppq.com/pytorch/pytorch/torch/csrc/jit/runtime/register_prim_ops.cpp.html#362

facebook-github-bot · 2020-07-03T06:10:40Z

This pull request has been merged in 824ab19.

Summary: Pull Request resolved: #40743 `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Imported from OSS Differential Revision: D22302151 fbshipit-source-id: 931000388e7501e9dd17bec2fad8a96b71a5efc5

… aten::repeat (#40933) * [quant][graphmode][fix] dequantize propagation for {add/mul}_scalar (#40596) Summary: Pull Request resolved: #40596 Previously the fusion patterns for {add/mul}_scalar is inconsistent since the op pattern produces a non-quantized tensor and the op replacement graph produces a quantized tensor Test Plan: Imported from OSS Differential Revision: D22251072 fbshipit-source-id: e16eb92cf6611578cca1ed8ebde961f8d0610137 * [quant][graphmode] Support quantization for `aten::apend` (#40743) Summary: Pull Request resolved: #40743 `aten::append` modifies input inplace and the output is ignored, these ops are not supported right now, so we'll need to first make `aten::append` non-inplace by change ``` ignored = aten::append(list, x) ``` to ``` x_list = aten::ListConstruct(x) result = aten::add(list, x_list) ``` and then quantize the aten::add instead. Test Plan: TestQuantizeJitOps.test_general_shape_ops Imported from OSS Differential Revision: D22302151 fbshipit-source-id: 931000388e7501e9dd17bec2fad8a96b71a5efc5

jerryzh168 requested a review from apaszke as a code owner June 29, 2020 23:03

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Jun 29, 2020

jerryzh168 requested review from raghuramank100, supriyar and vkuzo June 29, 2020 23:06

raghuramank100 reviewed Jun 29, 2020

View reviewed changes

jerryzh168 requested a review from raghuramank100 June 30, 2020 01:30

This was referenced Jul 1, 2020

[quant][graphmode][fix] Fold conv bn #40865

Closed

[quant][graphmode][fix] Print the node in error message #40889

Closed

vkuzo approved these changes Jul 2, 2020

View reviewed changes

raghuramank100 approved these changes Jul 2, 2020

View reviewed changes

facebook-github-bot closed this in 824ab19 Jul 3, 2020

facebook-github-bot added the merged label Jul 3, 2020

facebook-github-bot deleted the gh/jerryzh168/372/head branch July 6, 2020 14:18

jerryzh168 mentioned this pull request Jul 6, 2020

[quant][graphmode][fix] filter for list append change #41020

Closed

mruberry added the Merged label Oct 28, 2020

[quant][graphmode] Support quantization for aten::apend #40743

[quant][graphmode] Support quantization for aten::apend #40743

Uh oh!

Conversation

jerryzh168 commented Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

jerryzh168 commented Jun 29, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryzh168 Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryzh168 commented Jun 30, 2020

Uh oh!

vkuzo commented Jul 2, 2020

Uh oh!

vkuzo left a comment

Choose a reason for hiding this comment

Uh oh!

jerryzh168 commented Jul 2, 2020

Uh oh!

jerryzh168 commented Jul 2, 2020

Uh oh!

facebook-github-bot commented Jul 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[quant][graphmode] Support quantization for `aten::apend` #40743

[quant][graphmode] Support quantization for `aten::apend` #40743

jerryzh168 commented Jun 29, 2020 •

edited

Loading

dr-ci bot commented Jun 29, 2020 •

edited

Loading

jerryzh168 Jul 1, 2020 •

edited

Loading