Add CPU slow test job #73748

davidberard98 · 2022-03-03T18:39:58Z

Stack from ghstack:

-> Add CPU slow test job #73748

This adds CPU-only slow test jobs, which previously would never run.

Includes fixes for fx slow tests which were previously failing, but which never ran.

Differential Revision: D34628803

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) [ghstack-poisoned]

pytorch-bot · 2022-03-03T18:40:02Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/9accb66ede90993c9e9f5a2f8a72490b171a77b6/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
linux-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-manywheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
linux-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/trunk`	✅ triggered
linux-bionic-rocm4.5-py3.7	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/rocm`, `ciflow/trunk`	✅ triggered
linux-docs	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/docs`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-vulkan-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc5.4-mobile-lightweight-dispatch-build	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
macos-arm64-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
macos-arm64-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
macos-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
macos-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
macos-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
macos-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
windows-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
docker-builds	`ciflow/all`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/scheduled`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/scheduled`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/scheduled`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/scheduled`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`, `ciflow/trunk`	🚫 skipped
linux-docs-push	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-11-py3-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.3-py3.7-gcc7-debug	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.5-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
pytorch-xla-linux-bionic-py3.7-clang8	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`, `ciflow/xla`	🚫 skipped

facebook-github-bot · 2022-03-03T18:40:05Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/73748
↩️ [fb-only] Re-run with SSH instructions
Need help or want to give feedback on the CI? Visit our office hours

💊 CI failures summary and remediations

As of commit 5baacf0 (more details on the Dr. CI page):

1/1 failures introduced in this PR

1 failure not recognized by patterns:

Job	Step	Action
^{trunk / macos-11-py3-x86-64 / test (default, 1, 2, macos-11)}	^Unknown	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) [ghstack-poisoned]

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) ghstack-source-id: 0c9385d Pull Request resolved: #73748

davidberard98 · 2022-03-03T23:44:55Z

@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

davidberard98 · 2022-03-04T21:39:41Z

@albanD regarding slowTest on CPU, previously some of the CPU slowTests weren't running (e.g. TestNNCOpInfoCPU).

You can see that the runtime increased for the slow job:
On this PR (2.5 hrs): https://github.com/pytorch/pytorch/runs/5414065481?check_suite_focus=true
On a recent commit on master (1.5 hrs): https://github.com/pytorch/pytorch/runs/5424674750?check_suite_focus=true

Note: this also enables the CPU tests on slow-gradcheck jobs, should that be the case or would we only want to run it on the slow jobs?

albanD · 2022-03-07T14:58:57Z

Why does this change the slow gradcheck tests?

davidberard98 · 2022-03-07T17:59:43Z

@albanD not entirely sure what the slow-gradcheck tests do... but it seems like they're a subset of slow tests, in that PYTORCH_TEST_WITH_SLOW=1, PYTORCH_TEST_SKIP_FAST=1 are set. So if there were any slow-gradcheck tests marked @slowtest and which use instantiate_device_type_tests(), I think we'd want to enable the CPU versions? (otherwise they would never run, since there's no cpu-only slow-gradcheck jobs, either)

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) Differential Revision: [D34628803](https://our.internmc.facebook.com/intern/diff/D34628803) [ghstack-poisoned]

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) ghstack-source-id: 5afd37a Pull Request resolved: #73748

davidberard98 · 2022-03-09T01:25:54Z

@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

davidberard98 · 2022-03-09T01:27:16Z

actually, I'm going to disable the CPU tests on slow-gradcheck for now, since otherwise it times out.

malfet

Hmm, it feels like a much better solution would be to add slow CPU tests, wouldn't it? (which can be done by one-line change to https://github.com/pytorch/pytorch/blob/master/.github/scripts/generate_ci_workflows.py

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) Differential Revision: [D34628803](https://our.internmc.facebook.com/intern/diff/D34628803) [ghstack-poisoned]

This adds CPU-only slow test jobs, which previously would never run. Includes fixes for fx slow tests which were previously failing, but which never ran. Differential Revision: [D34628803](https://our.internmc.facebook.com/intern/diff/D34628803) [ghstack-poisoned]

This adds CPU-only slow test jobs, which previously would never run. Includes fixes for fx slow tests which were previously failing, but which never ran. ghstack-source-id: 27300a4 Pull Request resolved: #73748

malfet · 2022-03-21T23:57:58Z

.github/workflows/trunk.yml

+  linux-bionic-py3_7-clang9-slow-build:
+    name: linux-bionic-py3.7-clang9-slow
+    uses: pytorch/pytorch/.github/workflows/_linux-build.yml@master
+    with:
+      build-environment: linux-bionic-py3.7-clang9-slow
+      docker-image-name: pytorch-linux-bionic-py3.7-clang9


Do you need a separate build because there are no trunk-only builds?
@suo with new lint workflow is there a way to add a trunk-only test-shard to PR workflow?

it's just standard GH stuff now, so a job with an if statements would work.

Although: can we not just use one of the many trunk gcc builds instead? Is there something about clang that is special here

@suo how would the job with if statements work? e.g. where to put if statements?
re: gcc builds, I think we just want a cpu-only build. AFAICT all the other builds in trunk.yml are for cuda (other than parallelnative, which seems like a special configuration).

oh, that's interesting. Yeah it looks like there is no trunk-only CPU build. idk, personally I think this change is fine; I'd rather just do an extra build than make the workflows more marginally more complicated.

for reference, it would look like the same thing you have, with one additional line:

if: github.event == 'push'

But like I said, I feel like we should just keep pull jobs in pull.yml, and trunk jobs in trunk.yml. CPU builds take sub-10m and are super cheap, so it's not a big deal to overbuild a bit

davidberard98 · 2022-03-22T02:09:33Z

.github/workflows/trunk.yml

+  linux-bionic-py3_7-clang9-slow-build:
+    name: linux-bionic-py3.7-clang9-slow
+    uses: pytorch/pytorch/.github/workflows/_linux-build.yml@master
+    with:
+      build-environment: linux-bionic-py3.7-clang9-slow
+      docker-image-name: pytorch-linux-bionic-py3.7-clang9


@suo how would the job with if statements work? e.g. where to put if statements?
re: gcc builds, I think we just want a cpu-only build. AFAICT all the other builds in trunk.yml are for cuda (other than parallelnative, which seems like a special configuration).

davidberard98 · 2022-03-22T02:12:41Z

test/test_unary_ufuncs.py

    @onlyCPU
    @slowTest
    @dtypes(torch.float)
+    @unittest.skipIf(True, "Insufficient memory on linux.(2|4)xlarge")


@suo @malfet any opinions on whether we should skip this test entirely, or move the slow test to a 9xlarge just to have enough memory for this test?

davidberard98 · 2022-03-22T02:21:31Z

@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: Pull Request resolved: #73748 This adds CPU-only slow test jobs, which previously would never run. Includes fixes/skips for slow tests which fail (they need to be skipped now because they used to never run) Test Plan: Imported from OSS Reviewed By: malfet Differential Revision: D34628803 Pulled By: davidberard98 fbshipit-source-id: c090ab7bf7bda9e24ec5cdefa6fd35c6310dbac0

github-actions · 2022-03-23T21:18:16Z

Hey @davidberard98.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Pull Request resolved: #73748 This adds CPU-only slow test jobs, which previously would never run. Includes fixes/skips for slow tests which fail (they need to be skipped now because they used to never run) Test Plan: Imported from OSS Reviewed By: malfet Differential Revision: D34628803 Pulled By: davidberard98 fbshipit-source-id: c090ab7bf7bda9e24ec5cdefa6fd35c6310dbac0 (cherry picked from commit 06f7a94)

Enable CPU device_type tests on CUDA slow machines

9accb66

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) [ghstack-poisoned]

pytorch-bot bot added the ciflow/default label Mar 3, 2022

facebook-github-bot added the cla signed label Mar 3, 2022

davidberard98 added the ciflow/slow label Mar 3, 2022

Update on "Enable CPU device_type tests on CUDA slow machines"

42d0ef6

There are no CPU-only slow test machines, so this enables device_type slow tests (which previously would never run) [ghstack-poisoned]

davidberard98 requested a review from albanD March 4, 2022 21:35

albanD added the ciflow/slow-gradcheck label Mar 7, 2022

albanD requested a review from malfet March 7, 2022 14:59

davidberard98 requested a review from eellison March 10, 2022 21:30

malfet requested changes Mar 17, 2022

View reviewed changes

davidberard98 requested review from atalman, janeyx99 and seemethere as code owners March 17, 2022 22:08

davidberard98 changed the title ~~Enable CPU device_type tests on CUDA slow machines~~ Add CPU slow_test jobs Mar 17, 2022

davidberard98 removed request for IvanYashchuk, albanD, atalman, janeyx99, lezcano, nikitaved and seemethere March 21, 2022 20:33

davidberard98 added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 21, 2022

davidberard98 changed the title ~~Add CPU slow_test job~~ Add CPU slow test job Mar 21, 2022

davidberard98 removed ciflow/slow-gradcheck labels Mar 21, 2022

malfet approved these changes Mar 21, 2022

View reviewed changes

davidberard98 commented Mar 22, 2022

View reviewed changes

davidberard98 marked this pull request as ready for review March 22, 2022 15:38

suo removed the ciflow/default label Mar 22, 2022

pytorchmergebot closed this in 15c9870 Mar 23, 2022

davidberard98 added the topic: not user facing topic category label Mar 23, 2022

facebook-github-bot deleted the gh/davidberard98/56/head branch March 27, 2022 14:17

WBobby mentioned this pull request Aug 17, 2022

Add ROCm5.2.3/AMDGPU support for PyTorch WBobby/pytorch#2

Closed

Add CPU slow test job #73748

Add CPU slow test job #73748

Uh oh!

Conversation

davidberard98 commented Mar 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 3, 2022

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Mar 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

1 failure not recognized by patterns:

Uh oh!

davidberard98 commented Mar 3, 2022

Uh oh!

davidberard98 commented Mar 4, 2022

Uh oh!

albanD commented Mar 7, 2022

Uh oh!

davidberard98 commented Mar 7, 2022

Uh oh!

davidberard98 commented Mar 9, 2022

Uh oh!

davidberard98 commented Mar 9, 2022

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

suo Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

suo Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

suo Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 Mar 22, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 commented Mar 22, 2022

Uh oh!

github-actions bot commented Mar 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

davidberard98 commented Mar 3, 2022 •

edited

Loading

facebook-github-bot commented Mar 3, 2022 •

edited

Loading