fix mkldnn quantization issue for weight reorder error #86876
Conversation
[ghstack-poisoned]
XiaobingSuper
requested review from
digantdesai,
jerryzh168,
jianyuh,
kimishpatel,
salilsdesai and
z-a-f
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86876
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 5929579: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@jerryzh168 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@XiaobingSuper I have confirmed with my colleague that we can land this first, could you publish this draft PR? |
|
could you 1. clean up the fix 2. split the changes to default to a separate PR? |
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) [ghstack-poisoned]
github-actions
bot
added
module: cpu
Done, please help review it. |
jgong5
left a comment
jgong5
left a comment
There was a problem hiding this comment.
The change LGTM but could you please add the description the suspected issue and how the change addresses it?
|
@pytorchbot merge |
Merge failedReason: This PR has internal changes and must be landed via Phabricator Details for Dev Infra teamRaised by workflow job |
|
@jerryzh168 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@jerryzh168 , could you help see the internal failed test case? I can't access it. |
XiaobingSuper
added
the
intel priority
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) There has a potential issue when reordering block to block, now, if it failed, we will fist reorder to plain, and then reorder to the target block. cc jerryzh168 jianyuh raghuramank100 jamesr66a vkuzo jgong5 Xia-Weiwen leslie-fang-intel VitalyFedyunin mingfeima sanchitintel ashokei jingxu10 [ghstack-poisoned]
A test case is added. @@jerryzh168 |
|
Could we add some more context on what the issue was, which hardware the issue affects, and how this PR is fixing it? Also, to clarify, the test fails before this PR and passes after? |
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) For mkldnn quantization path, we will do weight prepack using dummy data to query the expected weight format, the packed weight's format may differ from the real input case(the weight format depends on the input's shape), and there will have a block weight to block weight reorder if the packed weight format differs with the expected weight format. The mkldnn may meet the following issue when doing such reorder(test on ICX machine): ``` test_conv_reorder_issue_onednn torch.ops.quantized.conv2d(qx, w_packed, output_scale=1.0, output_zero_point=0) File "/home/weiwen/.conda/envs/int8-dev/lib/python3.9/site-packages/torch/_ops.py", line 472, in __call__ return self._op(*args, **kwargs or {}) RuntimeError: could not create a primitive descriptor for a reorder primitive ``` This PR will fix it: if the block weight to block weight reorder is failed, we will reorder the block weight to plain weight first, and then reorder the plain weight to the target block weight. cc jerryzh168 jianyuh raghuramank100 jamesr66a vkuzo jgong5 Xia-Weiwen leslie-fang-intel VitalyFedyunin mingfeima sanchitintel ashokei jingxu10 [ghstack-poisoned]
@vkuzo , I add more context to descript the issue, please help see it. |
|
@jerryzh168, could we land it? |
|
sure, will try landing this again |
|
@jerryzh168 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@XiaobingSuper could you rebase the PR on master? |
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) For mkldnn quantization path, we will do weight prepack using dummy data to query the expected weight format, the packed weight's format may differ from the real input case(the weight format depends on the input's shape), and there will have a block weight to block weight reorder if the packed weight format differs with the expected weight format. The mkldnn may meet the following issue when doing such reorder(test on ICX machine): ``` test_conv_reorder_issue_onednn torch.ops.quantized.conv2d(qx, w_packed, output_scale=1.0, output_zero_point=0) File "/home/weiwen/.conda/envs/int8-dev/lib/python3.9/site-packages/torch/_ops.py", line 472, in __call__ return self._op(*args, **kwargs or {}) RuntimeError: could not create a primitive descriptor for a reorder primitive ``` This PR will fix it: if the block weight to block weight reorder is failed, we will reorder the block weight to plain weight first, and then reorder the plain weight to the target block weight. cc jerryzh168 jianyuh raghuramank100 jamesr66a vkuzo jgong5 Xia-Weiwen leslie-fang-intel VitalyFedyunin mingfeima sanchitintel ashokei jingxu10 [ghstack-poisoned]
rebased. |
|
@jerryzh168 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@jerryzh168 , please help land it if it is ok for you. |
|
@pytorchbot rebase |
|
@pytorchbot successfully started a rebase job. Check the current status here |
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) For mkldnn quantization path, we will do weight prepack using dummy data to query the expected weight format, the packed weight's format may differ from the real input case(the weight format depends on the input's shape), and there will have a block weight to block weight reorder if the packed weight format differs with the expected weight format. The mkldnn may meet the following issue when doing such reorder(test on ICX machine): ``` test_conv_reorder_issue_onednn torch.ops.quantized.conv2d(qx, w_packed, output_scale=1.0, output_zero_point=0) File "/home/weiwen/.conda/envs/int8-dev/lib/python3.9/site-packages/torch/_ops.py", line 472, in __call__ return self._op(*args, **kwargs or {}) RuntimeError: could not create a primitive descriptor for a reorder primitive ``` This PR will fix it: if the block weight to block weight reorder is failed, we will reorder the block weight to plain weight first, and then reorder the plain weight to the target block weight. cc jerryzh168 jianyuh raghuramank100 jamesr66a vkuzo jgong5 Xia-Weiwen leslie-fang-intel VitalyFedyunin mingfeima sanchitintel ashokei jingxu10 [ghstack-poisoned]
|
Successfully rebased |
trying to land, just need to make sure CI is green |
|
@jerryzh168 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Differential Revision: [D40351062](https://our.internmc.facebook.com/intern/diff/D40351062) For mkldnn quantization path, we will do weight prepack using dummy data to query the expected weight format, the packed weight's format may differ from the real input case(the weight format depends on the input's shape), and there will have a block weight to block weight reorder if the packed weight format differs with the expected weight format. The mkldnn may meet the following issue when doing such reorder(test on ICX machine): ``` test_conv_reorder_issue_onednn torch.ops.quantized.conv2d(qx, w_packed, output_scale=1.0, output_zero_point=0) File "/home/weiwen/.conda/envs/int8-dev/lib/python3.9/site-packages/torch/_ops.py", line 472, in __call__ return self._op(*args, **kwargs or {}) RuntimeError: could not create a primitive descriptor for a reorder primitive ``` This PR will fix it: if the block weight to block weight reorder is failed, we will reorder the block weight to plain weight first, and then reorder the plain weight to the target block weight. Pull Request resolved: pytorch#86876 Approved by: https://github.com/jgong5, https://github.com/jerryzh168
Stack from ghstack (oldest at bottom):
Differential Revision: D40351062
For mkldnn quantization path, we will do weight prepack using dummy data to query the expected weight format, the packed weight's format may differ from the real input case(the weight format depends on the input's shape), and there will have a block weight to block weight reorder if the packed weight format differs with the expected weight format. The mkldnn may meet the following issue when doing such reorder(test on ICX machine):
This PR will fix it: if the block weight to block weight reorder is failed, we will reorder the block weight to plain weight first, and then reorder the plain weight to the target block weight.
cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @VitalyFedyunin @mingfeima @sanchitintel @ashokei @jingxu10