Skip to content

Commit 73004bb

Browse files
vkuzovkuzo
committed
Update on "Adding a version serialization type to ConvPackedParam"
 This PR changes the format of `ConvPackedParam` in a nearly backwards-compatible way: * a new format is introduced which has more flexibility and a lower on-disk size * custom pickle functions are added to `ConvPackedParams` which know how to load the old format * the custom pickle functions are **not** BC because the output type of `__getstate__` has changed. We expect this to be acceptable as no user flows are actually broken (loading a v1 model with v2 code works), which is why we whitelist the failure. For a short forward compatibility period, we keep the `__getstate__` function with the old format, while `__getstate__` knows about both old and new format. The next PR in the stack will be landed some period after this PR, and will switch `__getstate__` to the new format. Note: the bc test cases were generated with the `v2` format, to allow manual testing of the `v2` code in this PR. Test plan: ``` // adhoc testing of saving v1 and loading in v2: https://gist.github.com/vkuzo/f3616c5de1b3109cb2a1f504feed69be // test that loading models with v1 conv params format works and leads to the same numerics python test/test_quantization.py TestSerialization.test_conv2d_graph python test/test_quantization.py TestSerialization.test_conv2d_nobias_graph // test that saving and loading models with v2 conv params format works and leads to same numerics python test/test_quantization.py TestSerialization.test_conv2d_graph_v2 python test/test_quantization.py TestSerialization.test_conv2d_nobias_graph_v2 // test that numerics on quantized MobileNetV2 do not change: // https://gist.github.com/vkuzo/d593385a88c30e22f46eba4248070c79 ``` Note: this is a newer copy of #40003 Differential Revision: [D23347832](https://our.internmc.facebook.com/intern/diff/D23347832) [ghstack-poisoned]
2 parents b4df205 + 5ef216b commit 73004bb

File tree

94 files changed

+3122
-665
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

94 files changed

+3122
-665
lines changed

‎.circleci/cimodel/data/binary_build_data.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,16 @@ def get_processor_arch_name(cuda_version):
5151
],
5252
)),
5353
# Skip CUDA-9.2 builds on Windows
54-
windows=([v for v in dimensions.CUDA_VERSIONS if v != '92'], OrderedDict(
55-
wheel=dimensions.STANDARD_PYTHON_VERSIONS,
56-
conda=dimensions.STANDARD_PYTHON_VERSIONS,
57-
libtorch=[
58-
"3.7",
59-
],
60-
)),
54+
windows=(
55+
[v for v in dimensions.CUDA_VERSIONS if v not in ['92', '110']],
56+
OrderedDict(
57+
wheel=dimensions.STANDARD_PYTHON_VERSIONS,
58+
conda=dimensions.STANDARD_PYTHON_VERSIONS,
59+
libtorch=[
60+
"3.7",
61+
],
62+
)
63+
),
6164
)
6265

6366
# GCC config variants:

‎.circleci/cimodel/data/dimensions.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"92",
66
"101",
77
"102",
8+
"110"
89
]
910

1011
STANDARD_PYTHON_VERSIONS = [

‎.circleci/cimodel/data/pytorch_build_data.py‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
]),
1111
("gcc", [
1212
("5.4", [ # All this subtree rebases to master and then build
13-
XImportant("3.6"),
1413
("3.6", [
14+
("important", [X(True)]),
1515
("parallel_tbb", [X(True)]),
1616
("parallel_native", [X(True)]),
1717
]),
@@ -69,7 +69,11 @@
6969
]),
7070
]),
7171
("gcc", [
72-
("9", [XImportant("3.8")]),
72+
("9", [
73+
("3.8", [
74+
("coverage", [XImportant(True)]),
75+
]),
76+
]),
7377
]),
7478
]),
7579
]
@@ -145,7 +149,8 @@ def child_constructor(self):
145149
"libtorch": LibTorchConfigNode,
146150
"important": ImportantConfigNode,
147151
"build_only": BuildOnlyConfigNode,
148-
"cuda_gcc_override": CudaGccOverrideConfigNode
152+
"cuda_gcc_override": CudaGccOverrideConfigNode,
153+
"coverage": CoverageConfigNode,
149154
}
150155
return next_nodes[experimental_feature]
151156

@@ -232,6 +237,15 @@ def child_constructor(self):
232237
return ExperimentalFeatureConfigNode
233238

234239

240+
class CoverageConfigNode(TreeConfigNode):
241+
242+
def init2(self, node_name):
243+
self.props["is_coverage"] = node_name
244+
245+
def child_constructor(self):
246+
return ExperimentalFeatureConfigNode
247+
248+
235249
class ImportantConfigNode(TreeConfigNode):
236250
def modify_label(self, label):
237251
return "IMPORTANT=" + str(label)

‎.circleci/cimodel/data/pytorch_build_definitions.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,8 +306,12 @@ def instantiate_configs():
306306
is_important = fc.find_prop("is_important") or False
307307
parallel_backend = fc.find_prop("parallel_backend") or None
308308
build_only = fc.find_prop("build_only") or False
309+
is_coverage = fc.find_prop("is_coverage") or False
309310
if build_only and restrict_phases is None:
310311
restrict_phases = ["build"]
312+
if is_coverage and restrict_phases is None:
313+
restrict_phases = ["build", "coverage_test"]
314+
311315

312316
gpu_resource = None
313317
if cuda_version and cuda_version != "10":

‎.circleci/cimodel/data/simple/docker_definitions.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
"pytorch-linux-xenial-py3.6-gcc7",
2929
"pytorch-linux-xenial-rocm3.3-py3.6",
3030
"pytorch-linux-xenial-rocm3.5.1-py3.6",
31+
"pytorch-linux-bionic-rocm3.7-py3.6",
3132
]
3233

3334

0 commit comments

Comments
 (0)