Skip to content

add warning when ParameterList/Dict is used with DataParallel #44405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
albanD wants to merge 9 commits into from
Closed

add warning when ParameterList/Dict is used with DataParallel #44405

albanD wants to merge 9 commits into from

Conversation

@albanD
Copy link
Collaborator

@albanD albanD commented Sep 9, 2020 •
edited
Loading

Stack from ghstack:

Differential Revision: D23783987

@albanD albanD requested a review from apaszke as a code owner September 9, 2020 18:40
albanD added a commit that referenced this pull request Sep 9, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
f5a2a32 
ghstack-source-id: da29155
Pull Request resolved: #44405
@albanD albanD requested a review from mruberry September 9, 2020 18:42
mruberry
mruberry reviewed Sep 9, 2020
View reviewed changes
mruberry
mruberry reviewed Sep 9, 2020
View reviewed changes
mruberry
mruberry reviewed Sep 9, 2020
View reviewed changes
@mruberry
Copy link
Collaborator

mruberry commented Sep 9, 2020

Would you like to add a test for the warning?

@dr-ci
Copy link

dr-ci bot commented Sep 9, 2020 •
edited
Loading

💊 CI failures summary and remediations

As of commit ec2d043 (more details on the Dr. CI page):

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 18 16:32:01 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 16:32:01 At: 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 16:32:01  
Sep 18 16:32:01 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 16:32:01  
Sep 18 16:32:01 At: 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 16:32:01  
Sep 18 16:32:01 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 16:32:01  
Sep 18 16:32:01 At: 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 16:32:01   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 16:32:01  
Sep 18 16:32:01 [W tensorpipe_agent.cpp:576] RPC agent for worker0 encountered error when reading incoming request from worker1: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 16:32:01 [W tensorpipe_agent.cpp:576] RPC agent for worker2 encountered error when reading incoming request from worker0: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 16:32:01 [W tensorpipe_agent.cpp:576] RPC agent for worker2 encountered error when reading incoming request from worker1: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 16:32:01 [W tensorpipe_agent.cpp:576] RPC agent for worker0 encountered error when reading incoming request from worker2: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 16:32:01 ok (1.840s)

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet:

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 40 times.

@albanD
Copy link
Collaborator Author

albanD commented Sep 9, 2020

@mruberry I am not sure where should I add such tests?

@mruberry
Copy link
Collaborator

mruberry commented Sep 9, 2020

@mruberry I am not sure where should I add such tests?

What about adding it to TestDataParallel?

pytorch/test/distributed/test_data_parallel.py

Line 22 in 364d03a

class TestDataParallel(TestCase):

@albanD
Copy link
Collaborator Author

albanD commented Sep 9, 2020

Ho I forgot that was a thing! Thanks.

ngimel
ngimel reviewed Sep 9, 2020
View reviewed changes
albanD added a commit that referenced this pull request Sep 9, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
8783465 
ghstack-source-id: 24a0a2d
Pull Request resolved: #44405
albanD added a commit that referenced this pull request Sep 10, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
12ce1dc 
ghstack-source-id: a2d8a06
Pull Request resolved: #44405
albanD added a commit that referenced this pull request Sep 10, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
d98f83a 
ghstack-source-id: 4b1c2d5
Pull Request resolved: #44405
albanD added a commit that referenced this pull request Sep 10, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
932a555 
ghstack-source-id: 68caa7a
Pull Request resolved: #44405
albanD added a commit that referenced this pull request Sep 12, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
7e753f0 
ghstack-source-id: c520945
Pull Request resolved: #44405
@codecov
Copy link

codecov bot commented Sep 12, 2020 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (gh/albanD/65/base@07b7e44). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@                 Coverage Diff                  @@
##             gh/albanD/65/base   #44405   +/-   ##
====================================================
  Coverage                     ?   67.98%           
====================================================
  Files                        ?      384           
  Lines                        ?    49581           
  Branches                     ?        0           
====================================================
  Hits                         ?    33707           
  Misses                       ?    15874           
  Partials                     ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07b7e44...ec2d043. Read the comment docs.

@albanD albanD requested review from mruberry and ngimel September 15, 2020 16:01
@mruberry mruberry removed the request for review from apaszke September 15, 2020 17:00
mruberry
mruberry reviewed Sep 15, 2020
View reviewed changes
test/test_nn.py Outdated
m = torch.nn.utils.weight_norm(m)
m = torch.nn.utils.weight_norm(m)

def test_weight_norm_parameterlistdict(self):
Copy link
Collaborator

@mruberry mruberry Sep 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this test more accurately verifies that parameter list and parameter dict throw warnings when a user tries to add attributes to them.

@mruberry mruberry self-requested a review September 15, 2020 23:50
mruberry
mruberry approved these changes Sep 15, 2020
View reviewed changes
Copy link
Collaborator

@mruberry mruberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngimel and I approve.

@ngimel: "It is strictly better than before."

@albanD albanD requested a review from rohan-varma as a code owner September 18, 2020 14:32
albanD added a commit that referenced this pull request Sep 18, 2020
@albanD
add warning when ParameterList/Dict is used with DataParallel
baaee6f 
ghstack-source-id: 4f87a41
Pull Request resolved: #44405
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Sep 22, 2020

@albanD merged this pull request in e155fbe.

@facebook-github-bot facebook-github-bot deleted the gh/albanD/65/head branch September 26, 2020 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mruberry mruberry mruberry approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@ngimel ngimel Awaiting requested review from ngimel

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

Assignees

No one assigned

Labels

Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@albanD @mruberry @facebook-github-bot @ngimel