onnx export of fake quantize functions #39738

skyw · 2020-06-09T17:28:35Z

As discussed in #39502.

This PR adds support for exporting fake_quantize_per_tensor_affine to a pair of QuantizeLinear and DequantizeLinear.

Exporting fake_quantize_per_channel_affine to ONNX depends on onnx/onnx#2772. will file another PR once ONNX merged the change.

It will generate ONNX graph like this:

@jamesr66a

torch/onnx/symbolic_opset10.py

test/onnx/test_onnx_opset.py

neginraoof · 2020-06-11T21:43:07Z

Thanks for adding the new op! Is there an example model using the FakeQuantize op? Would it be possible to have a model tests added in test_models.py?

skyw · 2020-06-12T00:15:40Z

Thanks for adding the new op! Is there an example model using the FakeQuantize op? Would it be possible to have a model tests added in test_models.py?

Yes, I can add one to test_models.py.
But I think the test will fail because test_models.py seems to test exporting on opset 9 by default. But QuantizeLinear and DequantizeLinear is added with opset 10. @neginraoof

dr-ci · 2020-06-12T00:54:46Z

💊 CI failures summary and remediations

As of commit 38e756b (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 40 times.

Pulling upstream

skyw · 2020-06-12T04:42:43Z

Thanks for adding the new op! Is there an example model using the FakeQuantize op? Would it be possible to have a model tests added in test_models.py?

Yes, I can add one to test_models.py.
But I think the test will fail because test_models.py seems to test exporting on opset 9 by default. But QuantizeLinear and DequantizeLinear is added with opset 10. @neginraoof

yeah, the test failed with RuntimeError: Exporting the operator fake_quantize_per_tensor_affine to ONNX opset version 9 is not supported. Support for this operator was added in version 10, try exporting with this version.
It shows a seg fault in later tests in CI. I tested on my local and got this error message. @neginraoof how should I proceed with the test? Is there a way to set using opset 10 in test_models.py? The test model works with torch.onnx.export()

neginraoof · 2020-06-15T16:30:01Z

So the purpose for test_models is to include a real model that uses FakeQuantize op.
For unit tests, I think onnxruntime test should cover this op.

But if you have a test that you want to disable for lower opsets, you might be able to do that using:
the skipIfUnsupportedMinOpsetVersion decorator.

from test_pytorch_common import skipIfUnsupportedMinOpsetVersion

skyw · 2020-06-15T21:47:18Z

Added skipIfUnsupportedMinOpsetVersion.

skyw · 2020-06-15T23:17:54Z

One test time out. It doesn't seem to be related to this PR. Do I need to worry about it? Clicked rerun but nothing happened.

Pull in upstream master

neginraoof

Looks good. Thanks!

skyw · 2020-06-18T20:32:17Z

@raghuramank100 @houseroad any comment?

torch/onnx/symbolic_opset10.py

spandantiwari · 2020-06-26T18:00:42Z

torch/onnx/symbolic_opset10.py

A minor suggestion - for code readers, it may be better to have a RuntimeError here instead of _unimplemented as this is something not supported in ONNX itself, and not something that is uncovered in the exporter.

@neginraoof suggested to use _unimplemented.

I agree with @neginraoof that it is better than ValueError that you had before. But _unimplemented is probably better suited for something that can be supported in the future. Since this is something not supported in ONNX (unless their is a spec change, which we can be considered), I feel that RuntimeError is a bit more appropriate.

changed to RuntimeError.

spandantiwari · 2020-06-26T18:06:35Z

test/onnx/model_defs/op_test.py

Would it be possible to add a model level test in _test_models_onnxruntime.py? It would also serve as documentation of the user scenario for this functionality.

Seems test_models_onnxruntime.py imports from test_pytorch_onnx_onnxruntime , https://github.com/pytorch/pytorch/blob/master/test/onnx/test_models_onnxruntime.py#L10

There is a test in test_pytorch_onnx_onnxruntime

That is just the run_model_test helper method. Also, the test point in test_pytorch_onnx_onnxruntime is unit test. I was suggesting adding a larger model test in this file. Do you have any model that required this functionality? That may be a good candidate for using for the model-level test.

There is a test added to test_models.py and run by test_models_onnxruntime.py.

Are you suggesting a larger one? Resnet50 feels too big for a test, maybe a LeNet?

Yes, we need a larger model test. We actually do have resnet50 as part of test_models.

Added a full qat resnet50.

pull in upstream master

gottbrath · 2020-07-01T20:44:50Z

@ngimel FYI
@raghuramank100 can you review this please?

Pull in upstream

skyw · 2020-07-14T02:21:24Z

Looks good. Sorry for being on this. Could you rebase to the master and retrigger the CI before landing?

rebased. CI passed

facebook-github-bot

@houseroad has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-07-16T06:10:37Z

@houseroad merged this pull request in 1fb2a7e.

facebook-github-bot · 2020-07-16T06:10:47Z

@houseroad merged this pull request in 1fb2a7e.

facebook-github-bot · 2020-07-16T06:10:57Z

@houseroad merged this pull request in 1fb2a7e.

LMerCy · 2020-07-17T02:30:21Z

Looks good. Sorry for being on this. Could you rebase to the master and retrigger the CI before landing?

rebased. CI passed

doese this feature support export quantized model to onnx? or just export to caffe2? @skyw

skyw · 2021-01-27T00:50:27Z

Looks good. Sorry for being on this. Could you rebase to the master and retrigger the CI before landing?

rebased. CI passed

doese this feature support export quantized model to onnx? or just export to caffe2? @skyw

Sorry I missed this...for 6 month
This one supports exporting fake quantized model in PYtorch to ONNX.

Summary: Fixes #39502 This PR adds support for exporting **fake_quantize_per_channel_affine** to a pair of QuantizeLinear and DequantizeLinear. Per tensor support was added by PR #39738. `axis` attribute of QuantizeLinear and DequantizeLinear, which is required for per channel support, is added in opset13 added by onnx/onnx#2772. [update 1/20/2021]: opset13 is being supported on master, the added function is now properly tested. Code also rebased to new master. The function is also tested offline with the following code ```python import torch from torch import quantization from torchvision import models qat_resnet18 = models.resnet18(pretrained=True).eval().cuda() qat_resnet18.qconfig = quantization.QConfig( activation=quantization.default_fake_quant, weight=quantization.default_per_channel_weight_fake_quant) quantization.prepare_qat(qat_resnet18, inplace=True) qat_resnet18.apply(quantization.enable_observer) qat_resnet18.apply(quantization.enable_fake_quant) dummy_input = torch.randn(16, 3, 224, 224).cuda() _ = qat_resnet18(dummy_input) for module in qat_resnet18.modules(): if isinstance(module, quantization.FakeQuantize): module.calculate_qparams() qat_resnet18.apply(quantization.disable_observer) qat_resnet18.cuda() input_names = [ "actual_input_1" ] output_names = [ "output1" ] torch.onnx.export(qat_resnet18, dummy_input, "quant_model.onnx", verbose=True, opset_version=13) ``` It can generate the desired graph. Pull Request resolved: #42835 Reviewed By: houseroad Differential Revision: D26293823 Pulled By: SplitInfinity fbshipit-source-id: 300498a2e24b7731b12fa2fbdea4e73dde80e7ea

Summary: Fixes #39502 This PR adds support for exporting **fake_quantize_per_channel_affine** to a pair of QuantizeLinear and DequantizeLinear. Per tensor support was added by PR #39738. `axis` attribute of QuantizeLinear and DequantizeLinear, which is required for per channel support, is added in opset13 added by onnx/onnx#2772. [update 1/20/2021]: opset13 is being supported on master, the added function is now properly tested. Code also rebased to new master. The function is also tested offline with the following code ```python import torch from torch import quantization from torchvision import models qat_resnet18 = models.resnet18(pretrained=True).eval().cuda() qat_resnet18.qconfig = quantization.QConfig( activation=quantization.default_fake_quant, weight=quantization.default_per_channel_weight_fake_quant) quantization.prepare_qat(qat_resnet18, inplace=True) qat_resnet18.apply(quantization.enable_observer) qat_resnet18.apply(quantization.enable_fake_quant) dummy_input = torch.randn(16, 3, 224, 224).cuda() _ = qat_resnet18(dummy_input) for module in qat_resnet18.modules(): if isinstance(module, quantization.FakeQuantize): module.calculate_qparams() qat_resnet18.apply(quantization.disable_observer) qat_resnet18.cuda() input_names = [ "actual_input_1" ] output_names = [ "output1" ] torch.onnx.export(qat_resnet18, dummy_input, "quant_model.onnx", verbose=True, opset_version=13) ``` It can generate the desired graph. Pull Request resolved: #42835 Reviewed By: houseroad Differential Revision: D26293823 Pulled By: SplitInfinity fbshipit-source-id: 300498a2e24b7731b12fa2fbdea4e73dde80e7ea Co-authored-by: Hao Wu <skyw@users.noreply.github.com>

pytorchbot added the open source label Jun 9, 2020

skyw mentioned this pull request Jun 9, 2020

Export fake quantization function to ONNX #39502

Closed

ngimel requested review from houseroad and raghuramank100 June 10, 2020 21:33

ngimel added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 10, 2020

neginraoof reviewed Jun 11, 2020

View reviewed changes

torch/onnx/symbolic_opset10.py Outdated Show resolved Hide resolved

neginraoof reviewed Jun 11, 2020

View reviewed changes

test/onnx/test_onnx_opset.py Outdated Show resolved Hide resolved

Merge pull request #1 from pytorch/master

0022f9f

Pulling upstream

Merge branch 'master' of https://github.com/pytorch/pytorch

879e048

skyw and others added 2 commits June 17, 2020 11:42

Merge pull request #2 from pytorch/master

5768be2

Pull in upstream master

Merge branch 'master' of https://github.com/skyw/pytorch

2062ac2

neginraoof approved these changes Jun 17, 2020

View reviewed changes

spandantiwari reviewed Jun 26, 2020

View reviewed changes

torch/onnx/symbolic_opset10.py Outdated Show resolved Hide resolved

spandantiwari reviewed Jun 26, 2020

View reviewed changes

skyw and others added 2 commits June 26, 2020 18:18

Merge pull request #3 from pytorch/master

1e1b135

pull in upstream master

Merge branch 'master' of https://github.com/skyw/pytorch

9bb9fb4

gottbrath requested review from ngimel and vkuzo July 1, 2020 20:43

gottbrath removed the request for review from ngimel July 1, 2020 20:45

Merge pull request #4 from pytorch/master

fb8eed7

Pull in upstream

skyw added 14 commits July 13, 2020 17:54

Add ONNX runtime test for fake_quantize_per_tensor_affine

454f3ca

Add ONNX opset test for fake_quantize_per_tensor_affine

e08789f

comment change

3eab237

use _unimplement for error raising

b48776c

remove test_fake_quantize_per_tensor in test_onnx_opset.py

2433d6e

Add fake quant net test in onnx/test_models.py

052a497

fit tests

d66f839

fix test_fake_quant in test_model.py

db21365

skip test_fake_quant in model test if opset version <10

b80dbca

fix typo

606c45a

add quant_resnet to onnx test_model.py

9255db3

change fake quant op boundary check to runtime error

14e276b

change test name

1b95333

commited wrong boundary check. fixing.

38e756b

facebook-github-bot reviewed Jul 15, 2020

View reviewed changes

facebook-github-bot closed this in 1fb2a7e Jul 16, 2020

facebook-github-bot added the merged label Jul 16, 2020

skyw mentioned this pull request Aug 10, 2020

onnx export of per channel fake quantize functions #42835

Closed

mruberry added the Merged label Oct 28, 2020

SplitInfinity mentioned this pull request Feb 18, 2021

onnx export of per channel fake quantize functions (#42835) #52430

Merged

onnx export of fake quantize functions #39738

onnx export of fake quantize functions #39738

Uh oh!

Conversation

skyw commented Jun 9, 2020

Uh oh!

Uh oh!

Uh oh!

neginraoof commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skyw commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

skyw commented Jun 12, 2020

Uh oh!

neginraoof commented Jun 15, 2020

Uh oh!

skyw commented Jun 15, 2020

Uh oh!

skyw commented Jun 15, 2020

Uh oh!

neginraoof left a comment

Choose a reason for hiding this comment

Uh oh!

skyw commented Jun 18, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skyw Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gottbrath commented Jul 1, 2020

Uh oh!

skyw commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 16, 2020

Uh oh!

facebook-github-bot commented Jul 16, 2020

Uh oh!

facebook-github-bot commented Jul 16, 2020

Uh oh!

LMerCy commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skyw commented Jan 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

neginraoof commented Jun 11, 2020 •

edited

Loading

skyw commented Jun 12, 2020 •

edited

Loading

dr-ci bot commented Jun 12, 2020 •

edited

Loading

skyw Jun 29, 2020 •

edited

Loading

skyw commented Jul 14, 2020 •

edited

Loading

LMerCy commented Jul 17, 2020 •

edited

Loading