Skip to content

[Gloo] Tests for Gloo Async Work Wait-level Timeouts #41265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
osalpekar wants to merge 6 commits into from
Closed

[Gloo] Tests for Gloo Async Work Wait-level Timeouts #41265

osalpekar wants to merge 6 commits into from

Conversation

@osalpekar
Copy link
Member

@osalpekar osalpekar commented Jul 10, 2020 •
edited
Loading

Stack from ghstack:

This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: D22470084

@osalpekar
[Gloo] Tests for Gloo Async Work Wait-level Timeouts
d11a27f 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
This was referenced Jul 10, 2020
Closed
Closed
Closed
Closed
Closed
osalpekar added a commit that referenced this pull request Jul 10, 2020
@osalpekar
[Gloo] Tests for Gloo Async Work Wait-level Timeouts
8109929 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

ghstack-source-id: 107533389
Pull Request resolved: #41265
@dr-ci
Copy link

dr-ci bot commented Jul 10, 2020 •
edited
Loading

💊 CI failures summary and remediations

As of commit 4fbf401 (more details on the Dr. CI page):

  • 4/4 failures possibly* introduced in this PR
    • 1/4 non-CircleCI failure(s)

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test (1/3)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jul 15 20:04:16 FAIL [0.032s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
Jul 15 20:04:14   test_sparse_default_std (__main__.TestNNInit) ... ok (0.008s) 
Jul 15 20:04:14   test_sparse_only_works_on_2d_inputs (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:04:15   test_trunc_normal (__main__.TestNNInit) ... ok (0.700s) 
Jul 15 20:04:16   test_uniform (__main__.TestNNInit) ... ok (0.898s) 
Jul 15 20:04:16   test_xavier_normal (__main__.TestNNInit) ... ok (0.106s) 
Jul 15 20:04:16   test_xavier_normal_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:04:16   test_xavier_uniform (__main__.TestNNInit) ... ok (0.080s) 
Jul 15 20:04:16   test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:04:16  
Jul 15 20:04:16 ====================================================================== 
Jul 15 20:04:16 FAIL [0.032s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
Jul 15 20:04:16 ---------------------------------------------------------------------- 
Jul 15 20:04:16 Traceback (most recent call last): 
Jul 15 20:04:16   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 777, in wrapper 
Jul 15 20:04:16     method(*args, **kwargs) 
Jul 15 20:04:16   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 777, in wrapper 
Jul 15 20:04:16     method(*args, **kwargs) 
Jul 15 20:04:16   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 674, in efail_fn_no_device 
Jul 15 20:04:16     return efail_fn(slf, None, *args, **kwargs) 
Jul 15 20:04:16   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 665, in efail_fn 
Jul 15 20:04:16     slf.fail('expected a non-deterministic error, but it was not raised')

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_test2 (2/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: expected a non-deterministic error, but it was not raised 
  test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.006s) 
 
====================================================================== 
FAIL [0.023s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 674, in efail_fn_no_device 
    return efail_fn(slf, None, *args, **kwargs) 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 665, in efail_fn 
    slf.fail('expected a non-deterministic error, but it was not raised') 
AssertionError: expected a non-deterministic error, but it was not raised 
 
---------------------------------------------------------------------- 
Ran 1800 tests in 671.536s 
 
FAILED (failures=1, skipped=90, expected failures=4) 
 
Generating XML reports... 
Generated XML report: test-reports\python-unittest\TEST-PackedSequenceTest-20200715200734.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAddRelu-20200715200734.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAvgPool-20200715200734.xml

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_test1 (3/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: expected a non-deterministic error, but it was not raised 
  test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.014s) 
 
====================================================================== 
FAIL [0.024s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 674, in efail_fn_no_device 
    return efail_fn(slf, None, *args, **kwargs) 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 665, in efail_fn 
    slf.fail('expected a non-deterministic error, but it was not raised') 
AssertionError: expected a non-deterministic error, but it was not raised 
 
---------------------------------------------------------------------- 
Ran 1800 tests in 716.828s 
 
FAILED (failures=1, skipped=90, expected failures=4) 
 
Generating XML reports... 
Generated XML report: test-reports\python-unittest\TEST-PackedSequenceTest-20200715193510.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAddRelu-20200715193510.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAvgPool-20200715193510.xml

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 14 times.

@osalpekar
Update on "[Gloo] Tests for Gloo Async Work Wait-level Timeouts"
560104e 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
osalpekar added a commit that referenced this pull request Jul 10, 2020
@osalpekar
[Gloo] Tests for Gloo Async Work Wait-level Timeouts
19bfef3 
Pull Request resolved: #41265

This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR
ghstack-source-id: 107555960

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)
@osalpekar
Update on "[Gloo] Tests for Gloo Async Work Wait-level Timeouts"
4641b8d 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
osalpekar added a commit that referenced this pull request Jul 13, 2020
@osalpekar
[Gloo] Tests for Gloo Async Work Wait-level Timeouts
3942333 
Pull Request resolved: #41265

This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR
ghstack-source-id: 107649144

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)
jiayisuse
jiayisuse reviewed Jul 15, 2020
View reviewed changes
torch/lib/c10d/test/ProcessGroupGlooTest.cpp Outdated
using namespace c10d::test;

constexpr auto kSendDelay = std::chrono::milliseconds(100);
constexpr auto kWaitDelay = std::chrono::milliseconds(1);
Copy link
Contributor

@jiayisuse jiayisuse Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: kWaitTimeout

@osalpekar
Update on "[Gloo] Tests for Gloo Async Work Wait-level Timeouts"
17ad75d 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
@osalpekar osalpekar requested a review from jiayisuse July 15, 2020 17:58
@osalpekar
Update on "[Gloo] Tests for Gloo Async Work Wait-level Timeouts"
bc5e14e 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
@osalpekar
Update on "[Gloo] Tests for Gloo Async Work Wait-level Timeouts"
4fbf401 
This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)

[ghstack-poisoned]
osalpekar added a commit that referenced this pull request Jul 15, 2020
@osalpekar
[Gloo] Tests for Gloo Async Work Wait-level Timeouts
be7e399 
Pull Request resolved: #41265

This PR adds tests for the Async Work wait-level timeouts that were added in the previous PR
ghstack-source-id: 107835732

Differential Revision: [D22470084](https://our.internmc.facebook.com/intern/diff/D22470084/)
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jul 16, 2020

This pull request has been merged in 81e9649.

@facebook-github-bot facebook-github-bot deleted the gh/osalpekar/59/head branch July 20, 2020 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jiayisuse jiayisuse jiayisuse approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@pietern pietern Awaiting requested review from pietern

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

Assignees

No one assigned

Labels

Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@osalpekar @facebook-github-bot @jiayisuse @mruberry