Skip to content

[1.2.0 branch] fix pin_memory_thread not exiting quickly #23647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
soumith merged 1 commit into from
Aug 1, 2019
Merged

[1.2.0 branch] fix pin_memory_thread not exiting quickly #23647

soumith merged 1 commit into from
Aug 1, 2019

Conversation

@ssnl
Copy link
Collaborator

@ssnl ssnl commented Aug 1, 2019

v1.2.0 branch version of #23646

@pytorchbot pytorchbot added the module: dataloader Related to torch.utils.data.DataLoader and Sampler label Aug 1, 2019
@ssnl ssnl changed the title fix pin_memory_thread not exiting quickly [1.2.0 branch] fix pin_memory_thread not exiting quickly Aug 1, 2019
@ssnl
Copy link
Collaborator Author

ssnl commented Aug 1, 2019

the asan build's ubsan check seems to be doing some weird things

Aug 01 06:51:29 + get_exit_code python -c 'import torch; torch._C._crash_if_csrc_ubsan(0)'
Aug 01 06:51:29 + set +e
Aug 01 06:51:29 + python -c 'import torch; torch._C._crash_if_csrc_ubsan(0)'
Aug 01 06:51:31 /var/lib/jenkins/workspace/torch/csrc/Module.cpp:151:18: runtime error: division by zero
Aug 01 06:51:31     #0 0x7f98296438b3 in THPModule_crashIfCsrcUBSAN(_object*, _object*) (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so+0x23658b3)
Aug 01 06:51:31     #1 0x55d90337d704 in _PyCFunction_FastCallDict /tmp/build/80754af9/python_1546130271559/work/Objects/methodobject.c:209
Aug 01 06:51:31     #2 0x55d90340442b in call_function /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4851
Aug 01 06:51:31     #3 0x55d903429389 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:3335
Aug 01 06:51:31     #4 0x55d9033ff288 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4166
Aug 01 06:51:31     #5 0x55d9033ff288 in PyEval_EvalCodeEx /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4187
Aug 01 06:51:31     #6 0x55d90340001b in PyEval_EvalCode /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:731
Aug 01 06:51:31     #7 0x55d9034823c3 in run_mod /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:1025
Aug 01 06:51:31     #8 0x55d90348245c in PyRun_StringFlags /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:949
Aug 01 06:51:31     #9 0x55d9034824be in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:445
Aug 01 06:51:31     #10 0x55d9034862c2 in run_command /tmp/build/80754af9/python_1546130271559/work/Modules/main.c:301
Aug 01 06:51:31     #11 0x55d9034862c2 in Py_Main /tmp/build/80754af9/python_1546130271559/work/Modules/main.c:749
Aug 01 06:51:31     #12 0x55d90334f02d in main /tmp/build/80754af9/python_1546130271559/work/Programs/python.c:69
Aug 01 06:51:31     #13 0x7f983d5f282f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291
Aug 01 06:51:31     #14 0x55d90342fe0d in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103
Aug 01 06:51:31 
Aug 01 06:51:31 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/torch/csrc/Module.cpp:151:18 in 
Aug 01 06:51:31 /var/lib/jenkins/workspace/torch/csrc/Module.cpp:152:31: runtime error: inf is outside the range of representable values of type 'int'
Aug 01 06:51:31     #0 0x7f98296438d5 in THPModule_crashIfCsrcUBSAN(_object*, _object*) (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so+0x23658d5)
Aug 01 06:51:31     #1 0x55d90337d704 in _PyCFunction_FastCallDict /tmp/build/80754af9/python_1546130271559/work/Objects/methodobject.c:209
Aug 01 06:51:31     #2 0x55d90340442b in call_function /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4851
Aug 01 06:51:31     #3 0x55d903429389 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:3335
Aug 01 06:51:31     #4 0x55d9033ff288 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4166
Aug 01 06:51:31     #5 0x55d9033ff288 in PyEval_EvalCodeEx /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:4187
Aug 01 06:51:31     #6 0x55d90340001b in PyEval_EvalCode /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:731
Aug 01 06:51:31     #7 0x55d9034823c3 in run_mod /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:1025
Aug 01 06:51:31     #8 0x55d90348245c in PyRun_StringFlags /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:949
Aug 01 06:51:31     #9 0x55d9034824be in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1546130271559/work/Python/pythonrun.c:445
Aug 01 06:51:31     #10 0x55d9034862c2 in run_command /tmp/build/80754af9/python_1546130271559/work/Modules/main.c:301
Aug 01 06:51:31     #11 0x55d9034862c2 in Py_Main /tmp/build/80754af9/python_1546130271559/work/Modules/main.c:749
Aug 01 06:51:31     #12 0x55d90334f02d in main /tmp/build/80754af9/python_1546130271559/work/Programs/python.c:69
Aug 01 06:51:31     #13 0x7f983d5f282f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291
Aug 01 06:51:31     #14 0x55d90342fe0d in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103
Aug 01 06:51:31 
Aug 01 06:51:31 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/torch/csrc/Module.cpp:152:31 in 
Aug 01 06:51:32 + retcode=0
Aug 01 06:51:32 + set -e
Aug 01 06:51:32 + return 0

@soumith soumith merged commit bb4ff00 into pytorch:v1.2.0 Aug 1, 2019
@ssnl ssnl deleted the dl_perf_120 branch August 1, 2019 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

module: dataloader Related to torch.utils.data.DataLoader and Sampler open source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ssnl @ezyang @soumith @pytorchbot