[quant][graph] Add support for FP16 dynamic quant #42222
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
This was referenced Jul 29, 2020
supriyar
added a commit
that referenced
this pull request
Jul 29, 2020
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 99716c4 Pull Request resolved: #42222
facebook-github-bot
added
the
oncall: jit
💊 CI failures summary and remediationsAs of commit 8016494 (more details on the Dr. CI page):
ci.pytorch.org: 1 failedThis comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 17 times. |
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
supriyar
added a commit
that referenced
this pull request
Jul 29, 2020
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: fdd1e38 Pull Request resolved: #42222
jerryzh168
reviewed
Jul 29, 2020
| } | ||
| if (quant_type == QuantType::DYNAMIC) { | ||
| if (isFP16NoopObserver(module, observer)) { | ||
| if (isFp16Observer(observer->input(0))) { |
Contributor
There was a problem hiding this comment.
can we just check dtype here?
Contributor
Author
There was a problem hiding this comment.
I think so, was just being more explicit by checking for the observer type as well.
jerryzh168
reviewed
Jul 29, 2020
| auto observer_module = module.attr(findObserverName(v).value()).toModule(); | ||
| return (observer_module.attr("dtype") == at::ScalarType::Half) && | ||
| isNoopObserver(observer); | ||
| bool isFp16Observer(Value* observer) { |
Contributor
There was a problem hiding this comment.
do we need this check? I think checking dtype is enough for our purposes?
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
supriyar
added a commit
that referenced
this pull request
Jul 30, 2020
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 0d270ed Pull Request resolved: #42222
jerryzh168
reviewed
Jul 30, 2020
jerryzh168
reviewed
Jul 30, 2020
jerryzh168
reviewed
Jul 30, 2020
Comment on lines
+2513
to
+2515
| observer_name = 'Fp16Observer = prim::GetAttr[name="_observer_' | ||
| FileCheck().check(observer_name) \ | ||
| .run(m.fc.graph) |
Contributor
There was a problem hiding this comment.
Looks like this check is not very useful, what do we want to check here?
Contributor
Author
There was a problem hiding this comment.
Just additional check for observer name match
jerryzh168
approved these changes
Jul 30, 2020
Contributor
jerryzh168
left a comment
jerryzh168
left a comment
There was a problem hiding this comment.
LG, had a few inline comments
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D22849220](https://our.internmc.facebook.com/intern/diff/D22849220) [ghstack-poisoned]
supriyar
added a commit
that referenced
this pull request
Jul 30, 2020
Summary: This change adds the necessary passes to perform FP16 dynamic quantization. We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights Test Plan: python test/test_quantization.py TestQuantizeJitOps Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: ce6e7dd Pull Request resolved: #42222
Contributor
|
This pull request has been merged in 6bd46b5. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary:
This change adds the necessary passes to perform FP16 dynamic quantization.
We skip inserting observers for activations based on the dtype (torch.float16) and only insert the Fp16Observer for weights
Test Plan:
python test/test_quantization.py TestQuantizeJitOps
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: D22849220