-
Notifications
You must be signed in to change notification settings - Fork 26.3k
[pkg] add generic ZipFile Reader/Writer #72237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
[ghstack-poisoned]
CI Flow Status⚛️ CI FlowRuleset - Version:
|
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit e089b8a (more details on the Dr. CI page):
🕵️ 1 new failure recognized by patternsThe following CI failures do not appear to be due to upstream breakages
|
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688) [ghstack-poisoned]
add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688) [ghstack-poisoned]
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
…stent_id to __reduce_package__ for storage" add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. This PR also adds changes in order to allow for torchscript model serialization without importing torch into torch.package.PackageImporter or torch.package.PackageExporter. Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688) [ghstack-poisoned]
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
can we split out the "add generic ZipFile Reader/Writer" and the "add change persistent_id to reduce_package for storage" changes? They seem unrelated and it would make things easier to review. |
suo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
back to you for splitting
…stent_id to __reduce_package__ for storage" add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. This PR also adds changes in order to allow for torchscript model serialization without importing torch into torch.package.PackageImporter or torch.package.PackageExporter. Differential Revision: [D33970688](https://our.internmc.facebook.com/intern/diff/D33970688) [ghstack-poisoned]
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
suo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure this PR achieves the outcome it intends to—we want to create an abstract ZipReader/Writer interface various clients can use, so that we can swap out the implementation for one backed by Python's zipfile module. But here, our clients are always expecting a TorchScriptZipFileWriter—if you pass it a different one, things will break.
In order to accomplish the goal of abstraction, all clients of the abstract ZipReader/Writer interface must only make use of functionality exposed by the interface.
|
Hey @PaliC. |
|
This pull request has been reverted by 00e2c14. To re-land this change, please open another pull request, assignthe same reviewers, fix the CI failures that caused the revert and make sure that the failing CI runs on the PR by applying the proper ciflow label (e.g., ciflow/trunk). |
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
1 similar comment
|
@PaliC has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Summary: Pull Request resolved: #72237 add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. https://www.internalfb.com/intern/diff/D35423079/ was reverted due to this refactor changing the name of where most of the implementation components of PackageExporter/PackageImporter come from like ModuleActionType_ etc. This diff also changes the import paths where these components come from to point to the correct file compared to D35423079 Test Plan: Imported from OSS Reviewed By: malfet Differential Revision: D35423079 Pulled By: PaliC fbshipit-source-id: 31abc4364d5fd007911cfb67cf36ebfac5d786f4
ghstack-source-id: d24c574 Pull Request resolved: pytorch/pytorch#72237
ghstack-source-id: 27c57fe Pull Request resolved: pytorch/pytorch#72237
ghstack-source-id: 9b14600 Pull Request resolved: pytorch/pytorch#72237
|
Unlanding this stack. Unfortunately it looks like this broke trunk / macos-11-py3-x86-64 / test (default, 1, 2, macos-11). See an example log here: https://github.com/pytorch/pytorch/runs/5863579163?check_suite_focus=true. |
|
This pull request has been reverted by 20be31d. To re-land this change, please open another pull request, assignthe same reviewers, fix the CI failures that caused the revert and make sure that the failing CI runs on the PR by applying the proper ciflow label (e.g., ciflow/trunk). |
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh` Test Plan: External CI Differential Revision: D35474261 fbshipit-source-id: 68a8203e12422ea394f1c597ea432d347ee6e48c
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh` Test Plan: External CI Differential Revision: D35474261 fbshipit-source-id: 0098fa87753293fc627ba4a95c5e03e8d14cf481
Summary: This PR is a response to pytorch#72237. It simply backouts of the revert diff Original Summaries: 1. add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. 2. Remove the requirement of torch in PackageExporter and PackageImporter. Now both of these modules work without torch, however, they can use torch for the necessary custom operations like in the case of torchscript and torch.fx. We do this by extending two endpoints similar to pickle call `persistent_id` and `__reduce_package__` Test Plan: External + Internal CI Also ran all package tests Differential Revision: D35474185 fbshipit-source-id: 47be5e95a1d5e37b938bd1aa9a8c1ee64721dc60
Summary: This PR is a response to pytorch#72237 and pytorch#74929 getting reverted as they were breaking MacOS testing builds due to the lack of the `zipfile` package. It does effectively the same thing. However, it also installs `zipfile` to MacOS builds in `.jenkins/pytorch/macos-common.sh` Test Plan: External CI Differential Revision: D35474261 fbshipit-source-id: 02687b7b690b1e9d922578b5c78f68f1cc323972
Summary: This PR is a response to pytorch#72237. It simply backouts of the revert diff Original Summaries: 1. add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs. 2. Remove the requirement of torch in PackageExporter and PackageImporter. Now both of these modules work without torch, however, they can use torch for the necessary custom operations like in the case of torchscript and torch.fx. We do this by extending two endpoints similar to pickle call `persistent_id` and `__reduce_package__` Test Plan: External + Internal CI Also ran all package tests Differential Revision: D35474185 fbshipit-source-id: 706a4f72f603f9aabc632c2c2b64ca0f2efd2d79
Stack from ghstack (oldest at bottom):
add a generic zip file reader/writer to torch.package in order to get rid of dependency on torch for non torchscript / tensor related usages of package. This also enables users to create a derived class from the zip file reader/writer classes to have their own serialization/deserialization if it's desired for performance needs.