Skip to content

Conversation

@NivekT
Copy link
Contributor

@NivekT NivekT commented Mar 9, 2022

Stack from ghstack:

Automatically generate datapipe.pyi via CMake and removing the generated .pyi file from Git. Users should have the .pyi file locally after building for the first time.

I will also be adding an internal equivalent diff for buck.

Differential Revision: D34868001

@pytorch-bot
Copy link

pytorch-bot bot commented Mar 9, 2022

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/a3f244cda038631f825149095657d1b2e52cd443/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows Labels (bold enabled) Status
Triggered Workflows
linux-binary-conda ciflow/binaries, ciflow/binaries_conda, ciflow/default ✅ triggered
linux-binary-libtorch-cxx11-abi ciflow/all, ciflow/binaries, ciflow/binaries_libtorch, ciflow/default, ciflow/trunk ✅ triggered
linux-binary-libtorch-pre-cxx11 ciflow/all, ciflow/binaries, ciflow/binaries_libtorch, ciflow/default, ciflow/trunk ✅ triggered
linux-binary-manywheel ciflow/all, ciflow/binaries, ciflow/binaries_wheel, ciflow/default, ciflow/trunk ✅ triggered
linux-bionic-py3.7-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/noarch, ciflow/trunk ✅ triggered
linux-bionic-rocm4.5-py3.7 ciflow/all, ciflow/default, ciflow/linux, ciflow/rocm, ciflow/trunk ✅ triggered
linux-docs ciflow/all, ciflow/cpu, ciflow/default, ciflow/docs, ciflow/linux, ciflow/trunk ✅ triggered
linux-vulkan-bionic-py3.7-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk, ciflow/vulkan ✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test ciflow/all, ciflow/bazel, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3-clang5-mobile-build ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile, ciflow/trunk ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile, ciflow/trunk ✅ triggered
linux-xenial-py3.7-clang7-asan ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/sanitizers, ciflow/trunk ✅ triggered
linux-xenial-py3.7-clang7-onnx ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/onnx, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc5.4-mobile-lightweight-dispatch-build ciflow/all, ciflow/cpu, ciflow/default, ciflow/libtorch, ciflow/linux, ciflow/mobile, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc7 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
linux-xenial-py3.7-gcc7-no-ops ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
macos-arm64-binary-conda ciflow/binaries, ciflow/binaries_conda, ciflow/default ✅ triggered
macos-arm64-binary-wheel ciflow/binaries, ciflow/binaries_wheel, ciflow/default ✅ triggered
macos-binary-conda ciflow/binaries, ciflow/binaries_conda, ciflow/default ✅ triggered
macos-binary-libtorch-cxx11-abi ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
macos-binary-libtorch-pre-cxx11 ciflow/binaries, ciflow/binaries_libtorch, ciflow/default ✅ triggered
macos-binary-wheel ciflow/binaries, ciflow/binaries_wheel, ciflow/default ✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single ciflow/all, ciflow/android, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit ciflow/all, ciflow/android, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/trunk ✅ triggered
win-vs2019-cpu-py3 ciflow/all, ciflow/cpu, ciflow/default, ciflow/trunk, ciflow/win ✅ triggered
win-vs2019-cuda11.3-py3 ciflow/all, ciflow/cuda, ciflow/default, ciflow/trunk, ciflow/win ✅ triggered
windows-binary-conda ciflow/binaries, ciflow/binaries_conda, ciflow/default ✅ triggered
windows-binary-libtorch-debug ciflow/all, ciflow/binaries, ciflow/binaries_libtorch, ciflow/default, ciflow/trunk ✅ triggered
windows-binary-libtorch-release ciflow/all, ciflow/binaries, ciflow/binaries_libtorch, ciflow/default, ciflow/trunk ✅ triggered
windows-binary-wheel ciflow/all, ciflow/binaries, ciflow/binaries_wheel, ciflow/default, ciflow/trunk ✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped
docker-builds ciflow/all, ciflow/trunk 🚫 skipped
ios-12-5-1-arm64 ciflow/all, ciflow/ios, ciflow/macos, ciflow/scheduled 🚫 skipped
ios-12-5-1-arm64-coreml ciflow/all, ciflow/ios, ciflow/macos, ciflow/scheduled 🚫 skipped
ios-12-5-1-arm64-custom-ops ciflow/all, ciflow/ios, ciflow/macos, ciflow/scheduled 🚫 skipped
ios-12-5-1-arm64-metal ciflow/all, ciflow/ios, ciflow/macos, ciflow/scheduled 🚫 skipped
ios-12-5-1-x86-64 ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
ios-12-5-1-x86-64-coreml ciflow/all, ciflow/ios, ciflow/macos, ciflow/trunk 🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/trunk 🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/trunk 🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow, ciflow/trunk 🚫 skipped
linux-docs-push ciflow/all, ciflow/cpu, ciflow/linux, ciflow/scheduled 🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops ciflow/all, ciflow/cuda, ciflow/linux, ciflow/trunk 🚫 skipped
macos-10-15-py3-arm64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
macos-11-py3-x86-64 ciflow/all, ciflow/macos, ciflow/trunk 🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled, ciflow/slow, ciflow/slow-gradcheck 🚫 skipped
periodic-linux-xenial-cuda11.3-py3.7-gcc7-debug ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-win-vs2019-cuda11.5-py3 ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win 🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build ciflow/all, ciflow/android, ciflow/cpu, ciflow/linux, ciflow/trunk 🚫 skipped
pytorch-xla-linux-bionic-py3.7-clang8 ciflow/all, ciflow/cpu, ciflow/linux, ciflow/trunk, ciflow/xla 🚫 skipped

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Mar 9, 2022

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 1fb559a (more details on the Dr. CI page):


💚 💚 Looks good so far! There are no failures yet. 💚 💚


This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

NivekT added a commit that referenced this pull request Mar 9, 2022
@NivekT NivekT changed the title [DataPipe] Automatically generate datapipe.pyi via CMake [BE Hackathon][DataPipe] Automatically generate datapipe.pyi via CMake Mar 9, 2022
@NivekT NivekT added module: data torch.utils.data release notes: dataloader release notes category labels Mar 9, 2022
@NivekT NivekT marked this pull request as draft March 9, 2022 23:02
Copy link
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The part about the build system update LGTM

You will most likely have to update your script to run fine from where it runs now (relative imports do help for these).

…yi via CMake"


Automatically generate `datapipe.pyi` via CMake

Since it will be automatically generated, I believe we should remove the output file from git VCS (like other generated .pyi file).

[ghstack-poisoned]
…yi via CMake"


Automatically generate `datapipe.pyi` via CMake

Since it will be automatically generated, I believe we should remove the output file from git VCS (like other generated .pyi file).

[ghstack-poisoned]
@NivekT NivekT marked this pull request as ready for review March 10, 2022 21:09
@NivekT NivekT requested a review from ejguan March 10, 2022 21:10
@NivekT
Copy link
Contributor Author

NivekT commented Mar 10, 2022

The part about the build system update LGTM

You will most likely have to update your script to run fine from where it runs now (relative imports do help for these).

I removed a dependency and I think it should work fine now

@NivekT NivekT requested a review from VitalyFedyunin March 10, 2022 21:11
…yi via CMake"


Automatically generate `datapipe.pyi` via CMake

Since it will be automatically generated, I believe we should remove the output file from git VCS (like other generated .pyi file).

[ghstack-poisoned]
NivekT added a commit that referenced this pull request Mar 10, 2022
@NivekT
Copy link
Contributor Author

NivekT commented Mar 10, 2022

@ejguan @albanD Should I be removing the generated .pyi file from git VCS?

Here is the trade off (should only impact PyTorch developers):

  1. Remove the file from Git - the .pyi will need be generated by running python setup.py develop or similar every time you checkout a new branch
  2. Keep the file in Git - when you go to a new branch, the Git version of the .pyi file will exist. When you run python setup.py develop and that will overwrite the Git version with the generated version. The Git version might be out of sync (generated version will overwrite the Git version).
    • We keep the file, should we add it to .gitignore or not?

I am currently keeping the .pyi file in Git but adding it to .gitignore. Maybe keeping it in Git and not in .gitignore is the best idea?

@NivekT NivekT requested a review from albanD March 14, 2022 17:42
@NivekT
Copy link
Contributor Author

NivekT commented Mar 14, 2022

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…yi via CMake"


Automatically generate `datapipe.pyi` via CMake and removing the generated .pyi file from Git. Users should have the .pyi file locally after building for the first time.

I will also be adding an internal equivalent diff for buck.

Differential Revision: [D34868001](https://our.internmc.facebook.com/intern/diff/D34868001)

[ghstack-poisoned]
NivekT added a commit that referenced this pull request Mar 14, 2022
@NivekT
Copy link
Contributor Author

NivekT commented Mar 14, 2022

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

…yi via CMake"


Automatically generate `datapipe.pyi` via CMake and removing the generated .pyi file from Git. Users should have the .pyi file locally after building for the first time.

I will also be adding an internal equivalent diff for buck.

Differential Revision: [D34868001](https://our.internmc.facebook.com/intern/diff/D34868001)

[ghstack-poisoned]
NivekT added a commit that referenced this pull request Mar 14, 2022
@NivekT
Copy link
Contributor Author

NivekT commented Mar 14, 2022

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot pushed a commit that referenced this pull request Mar 15, 2022
#73991)

Summary:
Pull Request resolved: #73991

Automatically generate `datapipe.pyi` via CMake and removing the generated .pyi file from Git. Users should have the .pyi file locally after building for the first time.

I will also be adding an internal equivalent diff for buck.

Test Plan: Imported from OSS

Reviewed By: ejguan

Differential Revision: D34868001

Pulled By: NivekT

fbshipit-source-id: 448c92da659d6b4c5f686407d3723933c266c74f
@github-actions
Copy link
Contributor

Hey @NivekT.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…p.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…p.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…p.py"


This should only land after pytorch/pytorch#73991

CI will fail until that PR is landed into Core's nightly.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…p.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 16, 2022
…p.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
@facebook-github-bot facebook-github-bot deleted the gh/nivekt/52/head branch March 19, 2022 14:17
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 21, 2022
…pipe.pyi in setup.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
NivekT added a commit to meta-pytorch/data that referenced this pull request Mar 21, 2022
…p.py"


This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044)

[ghstack-poisoned]
facebook-github-bot pushed a commit to meta-pytorch/data that referenced this pull request Mar 21, 2022
Summary:
Pull Request resolved: #290

This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control

Test Plan: Imported from OSS

Reviewed By: ejguan

Differential Revision: D34939044

Pulled By: NivekT

fbshipit-source-id: fa4cdcc01274ea8728b38b13a1410556ef85d8cb
@NivekT NivekT added the topic: not user facing topic category label Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed module: data torch.utils.data release notes: dataloader release notes category topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants