Skip to content

Conversation

@PaliC
Copy link
Contributor

@PaliC PaliC commented Feb 3, 2022

Stack from ghstack (oldest at bottom):

add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.

@pytorch-bot
Copy link

pytorch-bot bot commented Feb 3, 2022

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/b51247ad3ab4bd329fa4191f09a6e09c82795900/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows Labels (bold enabled) Status
Triggered Workflows
linux-binary-conda ciflow/binaries, ciflow/binaries_conda, ciflow/default ✅ triggered
linux-binary-libtorch-cxx11-abi ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
linux-binary-libtorch-pre-cxx11 ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
linux-binary-manywheel ciflow/binaries, ciflow/binaries_wheel, ciflow/default ✅ triggered
linux-bionic-py3.7-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/noarch, ciflow/trunk, ciflow/xla ✅ triggered
linux-bionic-rocm4.5-py3.7 ciflow/all, ciflow/default, ciflow/linux, ciflow/rocm, ciflow/trunk ✅ triggered
linux-docs ciflow/all, ciflow/cpu, ciflow/default, ciflow/docs, ciflow/linux, ciflow/trunk ✅ triggered
linux-vulkan-bionic-py3.7-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk, ciflow/vulkan ✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test ciflow/all, ciflow/bazel, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3-clang5-mobile-build ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile, ciflow/trunk ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile, ciflow/trunk ✅ triggered
linux-xenial-py3.7-clang7-asan ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/sanitizers, ciflow/trunk ✅ triggered
linux-xenial-py3.7-clang7-onnx ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/onnx, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc7 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc7-no-ops ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single ciflow/all, ciflow/android, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit ciflow/all, ciflow/android, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
win-vs2019-cpu-py3 ciflow/all, ciflow/cpu, ciflow/default, ciflow/trunk, ciflow/win ✅ triggered
win-vs2019-cuda11.3-py3 ciflow/all, ciflow/cuda, ciflow/default, ciflow/trunk, ciflow/win ✅ triggered
windows-binary-libtorch-cxx11-abi ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
windows-binary-libtorch-pre-cxx11 ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
windows-binary-wheel ciflow/binaries, ciflow/binaries_wheel, ciflow/default ✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped
docker-builds ciflow/all, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64 ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64-coreml ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64-custom-ops ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64-full-jit ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64-metal ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-x86-64 ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-x86-64-coreml ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-x86-64-full-jit ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/trunk 🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/trunk 🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow, ciflow/trunk 🚫 skipped
linux-docs-push ciflow/all, ciflow/cpu, ciflow/linux, ciflow/scheduled 🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops ciflow/all, ciflow/cuda, ciflow/linux, ciflow/trunk 🚫 skipped
macos-10-15-py3-arm64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
macos-11-py3-x86-64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled, ciflow/slow, ciflow/slow-gradcheck 🚫 skipped
periodic-linux-xenial-cuda11.1-py3.7-gcc7-debug ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-win-vs2019-cuda11.1-py3 ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win 🚫 skipped
periodic-win-vs2019-cuda11.5-py3 ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win 🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build ciflow/all, ciflow/android, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Feb 3, 2022

🔗 Helpful links

💊 CI failures summary and remediations

As of commit e089b8a (more details on the Dr. CI page):


  • 1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

See GitHub Actions build pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-07T13:43:02.5411905Z The PR is introduc...m to confirm whether this change is wanted or not.
2022-04-07T13:43:02.5400415Z processing existing schema:  text(__torch__.torch.classes.profiling.SourceRef _0) -> (str _0)
2022-04-07T13:43:02.5401237Z processing existing schema:  count(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-07T13:43:02.5402595Z processing existing schema:  duration_ns(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-07T13:43:02.5403925Z processing existing schema:  source(__torch__.torch.classes.profiling.SourceStats _0) -> (__torch__.torch.classes.profiling.SourceRef _0)
2022-04-07T13:43:02.5405614Z processing existing schema:  line_map(__torch__.torch.classes.profiling.SourceStats _0) -> (Dict(int, __torch__.torch.classes.profiling.InstructionStats) _0)
2022-04-07T13:43:02.5406387Z processing existing schema:  __init__(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-07T13:43:02.5408002Z processing existing schema:  enable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-07T13:43:02.5408991Z processing existing schema:  disable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-07T13:43:02.5410637Z processing existing schema:  _dump_stats(__torch__.torch.classes.profiling._ScriptProfile _0) -> (__torch__.torch.classes.profiling.SourceStats[] _0)
2022-04-07T13:43:02.5411633Z processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (NoneType _0)
2022-04-07T13:43:02.5411905Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2022-04-07T13:43:02.5411941Z 
2022-04-07T13:43:02.5412479Z Broken ops: [
2022-04-07T13:43:02.5412979Z 	aten::scatter_reduce_.two(Tensor(a!) self, int dim, Tensor index, Tensor src, str reduce, *, bool include_self=True) -> (Tensor(a!))
2022-04-07T13:43:02.5413050Z ]
2022-04-07T13:43:02.6346956Z + cleanup
2022-04-07T13:43:02.6347319Z + retcode=1
2022-04-07T13:43:02.6347607Z + set +x
2022-04-07T13:43:02.6387041Z ##[error]Process completed with exit code 1.
2022-04-07T13:43:02.6435835Z Prepare all required actions
2022-04-07T13:43:02.6436135Z Getting action download info

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Feb 3, 2022
@PaliC
Copy link
Contributor Author

PaliC commented Feb 3, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.

Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688)

[ghstack-poisoned]
@PaliC
Copy link
Contributor Author

PaliC commented Feb 3, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@PaliC PaliC changed the title [pkg] add generic ZipFile Reader/Writer [pkg] add generic ZipFile Reader/Writer + add change persistent_id to __reduce_package__ for storage Feb 3, 2022
…stent_id to __reduce_package__ for storage"


add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.

This PR also adds changes in order to allow for torchscript model serialization without importing torch into torch.package.PackageImporter or torch.package.PackageExporter.

Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688)

[ghstack-poisoned]
@PaliC
Copy link
Contributor Author

PaliC commented Feb 3, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@suo
Copy link
Member

suo commented Feb 4, 2022

can we split out the "add generic ZipFile Reader/Writer" and the "add change persistent_id to reduce_package for storage" changes? They seem unrelated and it would make things easier to review.

@suo suo modified the milestone: 1.11.0 Feb 4, 2022
Copy link
Member

@suo suo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

back to you for splitting

…stent_id to __reduce_package__ for storage"


add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.

This PR also adds changes in order to allow for torchscript model serialization without importing torch into torch.package.PackageImporter or torch.package.PackageExporter.

Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688)

[ghstack-poisoned]
@PaliC
Copy link
Contributor Author

PaliC commented Feb 9, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@PaliC PaliC changed the title [pkg] add generic ZipFile Reader/Writer + add change persistent_id to __reduce_package__ for storage [pkg] add generic ZipFile Reader/Writer Feb 9, 2022
Copy link
Member

@suo suo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this PR achieves the outcome it intends to—we want to create an abstract ZipReader/Writer interface various clients can use, so that we can swap out the implementation for one backed by Python's zipfile module. But here, our clients are always expecting a TorchScriptZipFileWriter—if you pass it a different one, things will break.

In order to accomplish the goal of abstraction, all clients of the abstract ZipReader/Writer interface must only make use of functionality exposed by the interface.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2022

Hey @PaliC.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

@PaliC PaliC reopened this Apr 6, 2022
@facebook-github-bot
Copy link
Contributor

This pull request has been reverted by 00e2c14. To re-land this change, please open another pull request, assignthe same reviewers, fix the CI failures that caused the revert and make sure that the failing CI runs on the PR by applying the proper ciflow label (e.g., ciflow/trunk).

@PaliC
Copy link
Contributor Author

PaliC commented Apr 6, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

1 similar comment
@PaliC
Copy link
Contributor Author

PaliC commented Apr 6, 2022

@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot pushed a commit that referenced this pull request Apr 6, 2022
Summary:
Pull Request resolved: #72237

add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.

https://www.internalfb.com/intern/diff/D35423079/ was reverted due to this refactor changing the name of where most of the implementation components of PackageExporter/PackageImporter come from like ModuleActionType_ etc.
This diff also changes the import paths where these components come from to point to the correct file compared to D35423079

Test Plan: Imported from OSS

Reviewed By: malfet

Differential Revision: D35423079

Pulled By: PaliC

fbshipit-source-id: 31abc4364d5fd007911cfb67cf36ebfac5d786f4
@malfet malfet closed this in d4a709b Apr 6, 2022
NesrineMHB pushed a commit to NesrineMHB/pytorch that referenced this pull request Apr 7, 2022
ghstack-source-id: d24c574
Pull Request resolved: pytorch/pytorch#72237
NesrineMHB pushed a commit to NesrineMHB/pytorch that referenced this pull request Apr 7, 2022
ghstack-source-id: 27c57fe
Pull Request resolved: pytorch/pytorch#72237
NesrineMHB pushed a commit to NesrineMHB/pytorch that referenced this pull request Apr 7, 2022
ghstack-source-id: 9b14600
Pull Request resolved: pytorch/pytorch#72237
@mruberry
Copy link
Collaborator

mruberry commented Apr 7, 2022

Unlanding this stack. Unfortunately it looks like this broke trunk / macos-11-py3-x86-64 / test (default, 1, 2, macos-11). See an example log here: https://github.com/pytorch/pytorch/runs/5863579163?check_suite_focus=true.

@mruberry mruberry reopened this Apr 7, 2022
@facebook-github-bot
Copy link
Contributor

This pull request has been reverted by 20be31d. To re-land this change, please open another pull request, assignthe same reviewers, fix the CI failures that caused the revert and make sure that the failing CI runs on the PR by applying the proper ciflow label (e.g., ciflow/trunk).

@PaliC PaliC closed this Apr 7, 2022
PaliC added a commit to PaliC/pytorch that referenced this pull request Apr 7, 2022
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also  installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh`

Test Plan: External CI

Differential Revision: D35474261

fbshipit-source-id: 68a8203e12422ea394f1c597ea432d347ee6e48c
PaliC pushed a commit to PaliC/pytorch that referenced this pull request Apr 7, 2022
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also  installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh`

Test Plan: External CI

Differential Revision: D35474261

fbshipit-source-id: 0098fa87753293fc627ba4a95c5e03e8d14cf481
PaliC added a commit to PaliC/pytorch that referenced this pull request Apr 7, 2022
Summary:
This PR is a response to pytorch#72237. It simply backouts of the revert diff

Original Summaries:
1. add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.
2. Remove the requirement of torch in PackageExporter and PackageImporter. Now both of these modules work without torch, however, they can use torch for the necessary custom operations like in the case of torchscript and torch.fx. We do this by extending two endpoints similar to pickle call `persistent_id` and `__reduce_package__`

Test Plan:
External + Internal CI
Also ran all package tests

Differential Revision: D35474185

fbshipit-source-id: 47be5e95a1d5e37b938bd1aa9a8c1ee64721dc60
PaliC pushed a commit to PaliC/pytorch that referenced this pull request Apr 7, 2022
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also  installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh`

Test Plan: External CI

Differential Revision: D35474261

fbshipit-source-id: 02687b7b690b1e9d922578b5c78f68f1cc323972
PaliC pushed a commit to PaliC/pytorch that referenced this pull request Apr 7, 2022
Summary:
This PR is a response to pytorch#72237. It simply backouts of the revert diff

Original Summaries:
1. add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.
2. Remove the requirement of torch in PackageExporter and PackageImporter. Now both of these modules work without torch, however, they can use torch for the necessary custom operations like in the case of torchscript and torch.fx. We do this by extending two endpoints similar to pickle call `persistent_id` and `__reduce_package__`

Test Plan:
External + Internal CI
Also ran all package tests

Differential Revision: D35474185

fbshipit-source-id: 706a4f72f603f9aabc632c2c2b64ca0f2efd2d79
@facebook-github-bot facebook-github-bot deleted the gh/PaliC/133/head branch April 11, 2022 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed oncall: jit Add this issue/PR to JIT oncall triage queue Reverted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants