quantization: autogenerate quantization backend configs for documentation #75126

vkuzo · 2022-04-01T19:18:45Z

Stack from ghstack:

quantization: autogenerate quantization backend configs for documentation #75126 quantization: autogenerate quantization backend configs for documentation

Summary:

Quantization has a high volume of configurations of how to quantize an
op for a reference model representation which is useful for a lowering
step for a backend. An example of this is

 {'dtype_configs': [{'input_dtype': torch.quint8,
										 'output_dtype': torch.quint8}],
	'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>,
	'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>},

These configs are checked into master, and they are created with Python functions.
Therefore, there is no easy way for the user to see what the configs actually
are without running some Python code.

This PR is one approach to document these configs. Here is what this is doing:

during documentation build, write a text file of the configs
render that text file on a quantization page, with some additional context

In the future, this could be extended to autogenerate better looking tables
such as: op support per backend and dtype, op support per valid quantization settings per backend,
etc.

Test plan:

cd docs
make html
cd html
python -m http.server 8000
// render http://[::]:8000/quantization-backend-configuration.html
// it renders correctly

Differential Revision: D35365461

…umentation Summary: Quantization has a high volume of configurations of how to quantize an op for a reference model representation which is useful for a lowering step for a backend. An example of this is ``` {'dtype_configs': [{'input_dtype': torch.quint8, 'output_dtype': torch.quint8}], 'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>, 'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>}, ``` These configs are checked into master, and they are created with Python functions. Therefore, there is no easy way for the user to see what the configs actually are without running some Python code. This PR is one approach to document these configs. Here is what this is doing: 1. during documentation build, write a text file of the configs 2. render that text file on a quantization page, with some additional context In the future, this could be extended to autogenerate better looking tables such as: op support per backend and dtype, op support per valid quantization settings per backend, etc. Currently this is an RFC to ensure there aren't any security concerns with rendering a text file. If there aren't, this could move to review by quantization team. Note: the content being rendered is a placeholder, for now. We need a better looking print-out, with more context, before we can ship something like this to users. Test plan: ``` cd docs make html cd html python -m http.server 8000 // render http://[::]:8000/quantization-backend-configuration.html // it renders correctly ``` [ghstack-poisoned]

facebook-github-bot · 2022-04-01T19:18:50Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/75126
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours

💊 CI failures summary and remediations

As of commit 169bdb5 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

…umentation Summary: Quantization has a high volume of configurations of how to quantize an op for a reference model representation which is useful for a lowering step for a backend. An example of this is ``` {'dtype_configs': [{'input_dtype': torch.quint8, 'output_dtype': torch.quint8}], 'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>, 'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>}, ``` These configs are checked into master, and they are created with Python functions. Therefore, there is no easy way for the user to see what the configs actually are without running some Python code. This PR is one approach to document these configs. Here is what this is doing: 1. during documentation build, write a text file of the configs 2. render that text file on a quantization page, with some additional context In the future, this could be extended to autogenerate better looking tables such as: op support per backend and dtype, op support per valid quantization settings per backend, etc. Currently this is an RFC to ensure there aren't any security concerns with rendering a text file. If there aren't, this could move to review by quantization team. Note: the content being rendered is a placeholder, for now. We need a better looking print-out, with more context, before we can ship something like this to users. Test plan: ``` cd docs make html cd html python -m http.server 8000 // render http://[::]:8000/quantization-backend-configuration.html // it renders correctly ``` ghstack-source-id: 9227148 Pull Request resolved: #75126

andrewor14

By the way I just landed a bunch of new configs. Could you rebase and see what it looks like now?

andrewor14 · 2022-04-04T15:00:39Z

docs/source/quantization-backend-configuration.rst

+
+  from torch.ao.quantization.fx.backend_config import get_native_backend_config_dict
+  from pprint import pprint
+  pprint(get_native_backend_config_dict())


I think this would look better:

import json print(json.dumps(get_native_backend_config_dict(), indent=4))

Something like this would work:

from torch.utils._pytree import tree_map config = get_native_backend_config_dict() # convert Python types to strings, so json module can understand them config_str = tree_map(str, config) f.write(json.dumps(config_str, indent=2))

However, the results would look like:

{ "pattern": "<function leaky_relu at 0x7fd4083abb80>", "observation_type": "ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT", "dtype_configs": [ { "input_dtype": "torch.quint8", "output_dtype": "torch.quint8" } ] },

It's easier to read, but also everything is printed as strings, which may be confusing to the user since these objects are not actually strings. I think a custom printing function may have to be written to both make it nice on the eyes and also not print incorrect types.

I see. Maybe we can do that in the future then

docs/source/scripts/build_quantization_configs.py

…r documentation" Summary: Quantization has a high volume of configurations of how to quantize an op for a reference model representation which is useful for a lowering step for a backend. An example of this is ``` {'dtype_configs': [{'input_dtype': torch.quint8, 'output_dtype': torch.quint8}], 'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>, 'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>}, ``` These configs are checked into master, and they are created with Python functions. Therefore, there is no easy way for the user to see what the configs actually are without running some Python code. This PR is one approach to document these configs. Here is what this is doing: 1. during documentation build, write a text file of the configs 2. render that text file on a quantization page, with some additional context In the future, this could be extended to autogenerate better looking tables such as: op support per backend and dtype, op support per valid quantization settings per backend, etc. Test plan: ``` cd docs make html cd html python -m http.server 8000 // render http://[::]:8000/quantization-backend-configuration.html // it renders correctly ``` [ghstack-poisoned]

…umentation Summary: Quantization has a high volume of configurations of how to quantize an op for a reference model representation which is useful for a lowering step for a backend. An example of this is ``` {'dtype_configs': [{'input_dtype': torch.quint8, 'output_dtype': torch.quint8}], 'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>, 'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>}, ``` These configs are checked into master, and they are created with Python functions. Therefore, there is no easy way for the user to see what the configs actually are without running some Python code. This PR is one approach to document these configs. Here is what this is doing: 1. during documentation build, write a text file of the configs 2. render that text file on a quantization page, with some additional context In the future, this could be extended to autogenerate better looking tables such as: op support per backend and dtype, op support per valid quantization settings per backend, etc. Currently this is an RFC to ensure there aren't any security concerns with rendering a text file. If there aren't, this could move to review by quantization team. Note: the content being rendered is a placeholder, for now. We need a better looking print-out, with more context, before we can ship something like this to users. Test plan: ``` cd docs make html cd html python -m http.server 8000 // render http://[::]:8000/quantization-backend-configuration.html // it renders correctly ``` ghstack-source-id: f69b752 Pull Request resolved: #75126

vkuzo · 2022-04-04T17:48:37Z

@vkuzo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…tion (#75126) Summary: Pull Request resolved: #75126 Quantization has a high volume of configurations of how to quantize an op for a reference model representation which is useful for a lowering step for a backend. An example of this is ``` {'dtype_configs': [{'input_dtype': torch.quint8, 'output_dtype': torch.quint8}], 'observation_type': <ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT: 0>, 'pattern': <class 'torch.nn.modules.conv.ConvTranspose1d'>}, ``` These configs are checked into master, and they are created with Python functions. Therefore, there is no easy way for the user to see what the configs actually are without running some Python code. This PR is one approach to document these configs. Here is what this is doing: 1. during documentation build, write a text file of the configs 2. render that text file on a quantization page, with some additional context In the future, this could be extended to autogenerate better looking tables such as: op support per backend and dtype, op support per valid quantization settings per backend, etc. Test Plan: ``` cd docs make html cd html python -m http.server 8000 // render http://[::]:8000/quantization-backend-configuration.html // it renders correctly ``` Reviewed By: ejguan Differential Revision: D35365461 Pulled By: vkuzo fbshipit-source-id: d60f776ccb57da9db3d09550e4b27bd5e725635a

github-actions · 2022-04-04T22:23:08Z

Hey @vkuzo.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot added the cla signed label Apr 1, 2022

vkuzo changed the title ~~[rfc] quantization: autogenerate quantization backend configs for documentation~~ quantization: autogenerate quantization backend configs for documentation Apr 4, 2022

vkuzo requested review from andrewor14 and jerryzh168 April 4, 2022 14:35

andrewor14 approved these changes Apr 4, 2022

View reviewed changes

jerryzh168 reviewed Apr 4, 2022

View reviewed changes

docs/source/scripts/build_quantization_configs.py Outdated Show resolved Hide resolved

jerryzh168 approved these changes Apr 4, 2022

View reviewed changes

pytorchmergebot closed this in 74b23b2 Apr 4, 2022

facebook-github-bot deleted the gh/vkuzo/482/head branch April 8, 2022 14:17

WBobby mentioned this pull request Aug 17, 2022

Add ROCm5.2.3/AMDGPU support for PyTorch WBobby/pytorch#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

quantization: autogenerate quantization backend configs for documentation #75126

quantization: autogenerate quantization backend configs for documentation #75126

Uh oh!

vkuzo commented Apr 1, 2022 •

edited

Loading

Uh oh!

facebook-github-bot commented Apr 1, 2022 •

edited

Loading

Uh oh!

andrewor14 left a comment

Uh oh!

andrewor14 Apr 4, 2022

Uh oh!

vkuzo Apr 4, 2022

Uh oh!

andrewor14 Apr 4, 2022

Uh oh!

Uh oh!

vkuzo commented Apr 4, 2022

Uh oh!

github-actions bot commented Apr 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

quantization: autogenerate quantization backend configs for documentation #75126

quantization: autogenerate quantization backend configs for documentation #75126

Uh oh!

Conversation

vkuzo commented Apr 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Apr 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

Uh oh!

andrewor14 left a comment

Choose a reason for hiding this comment

Uh oh!

andrewor14 Apr 4, 2022

Choose a reason for hiding this comment

Uh oh!

vkuzo Apr 4, 2022

Choose a reason for hiding this comment

Uh oh!

andrewor14 Apr 4, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vkuzo commented Apr 4, 2022

Uh oh!

github-actions bot commented Apr 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vkuzo commented Apr 1, 2022 •

edited

Loading

facebook-github-bot commented Apr 1, 2022 •

edited

Loading