Skip to content

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize()#20944

Merged
vstinner merged 1 commit into
python:masterfrom
vstinner:skip_if_broken_multiprocessing_synchronize
Jun 18, 2020
Merged

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize()#20944
vstinner merged 1 commit into
python:masterfrom
vstinner:skip_if_broken_multiprocessing_synchronize

Conversation

@vstinner

@vstinner vstinner commented Jun 17, 2020 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.

https://bugs.python.org/issue38377

@vstinner
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize()
0edba4a 
On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.
@vstinner

vstinner commented Jun 17, 2020

Copy link
Copy Markdown
Member Author

Maybe my function is too strict, not all tests require to create a SemLock. But my intent is to avoid coupling tests like test_asyncio with low-level implementation details of multiprocessing.

In practice, only the OpenSUSE build farm should be affected by this change. In general, /dev/shm/ is writable by anyone. Example on Fedora 32:

$ ls -ld /dev/shm/
drwxrwxrwt. 2 root root 40 17 juin  18:13 /dev/shm//

@vstinner

vstinner commented Jun 17, 2020

Copy link
Copy Markdown
Member Author

@mcepl: Would it be possible for you to check if my change fix https://bugs.python.org/issue38377 for your use case?

@vstinner vstinner mentioned this pull request Jun 17, 2020
Closed
@vstinner

vstinner commented Jun 17, 2020

Copy link
Copy Markdown
Member Author

cc @pitrou

@mcepl

mcepl commented Jun 17, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor

On Python 3.8:

[ 1064s] test test_concurrent_futures crashed -- Traceback (most recent call last):
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/libregrtest/runtest.py", line 270, in _runtest_inner
[ 1064s]     refleak = _runtest_inner2(ns, test_name)
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/libregrtest/runtest.py", line 221, in _runtest_inner2
[ 1064s]     the_module = importlib.import_module(abstest)
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/importlib/__init__.py", line 127, in import_module
[ 1064s]     return _bootstrap._gcd_import(name[level:], package, level)
[ 1064s]   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[ 1064s]   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[ 1064s]   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[ 1064s]   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[ 1064s]   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
[ 1064s]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/test_concurrent_futures.py", line 6, in <module>
[ 1064s]     support.skip_if_broken_multiprocessing_synchronize()
[ 1064s] NameError: name 'support' is not defined

Replacing support. with test.support. made it work.

@vstinner

vstinner commented Jun 17, 2020

Copy link
Copy Markdown
Member Author

[ 1064s] NameError: name 'support' is not defined

Well, that's a bug in your backport.

Replacing support. with test.support. made it work.

I understand that this PR fix https://bugs.python.org/issue38377, good!

@mcepl

mcepl commented Jun 17, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor

I understand that this PR fix https://bugs.python.org/issue38377, good!

Yes, it does. Pushing it now to the official openSUSE packages (OBS_dev-shm.patch)

@vstinner

vstinner commented Jun 17, 2020

Copy link
Copy Markdown
Member Author

@mcepl: So which approach do you prefer? This PR or your PR #19073?

@mcepl

mcepl commented Jun 18, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor

Obviously this one, it is a way more complete and less hassle to everybody. I’ll close my PR.

@vstinner
vstinner merged commit ddbeb2f into python:master Jun 18, 2020
@vstinner
vstinner deleted the skip_if_broken_multiprocessing_synchronize branch June 18, 2020 12:53
@vstinner

vstinner commented Jun 18, 2020

Copy link
Copy Markdown
Member Author

Obviously this one, it is a way more complete and less hassle to everybody. I’ll my PR.

Ok, I merged my PR and I'm backporting it to 3.8 and 3.9.

vstinner added a commit that referenced this pull request Jun 18, 2020
@vstinner
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (G…
b1e7361 
…H-20944) (GH-20962)

On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.

(cherry picked from commit ddbeb2f)
@bedevere-bot

bedevere-bot commented Jun 18, 2020

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.9 has failed when building commit b1e7361.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/696/builds/119) and take a look at the build logs.
  4. Check if the failure is related to this commit (b1e7361) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/696/builds/119

Failed tests:

  • test_multiprocessing_forkserver

Failed subtests:

  • test_resource_tracker_reused - test.test_multiprocessing_forkserver.TestResourceTracker

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

406 tests OK.

10 slowest tests:

  • test_concurrent_futures: 4 min 25 sec
  • test_tokenize: 3 min 5 sec
  • test_unparse: 3 min 1 sec
  • test_multiprocessing_spawn: 2 min 41 sec
  • test_multiprocessing_forkserver: 2 min 2 sec
  • test_lib2to3: 2 min 1 sec
  • test_unicodedata: 1 min 49 sec
  • test_asyncio: 1 min 44 sec
  • test_capi: 1 min 44 sec
  • test_pickle: 1 min 4 sec

1 test failed:
test_multiprocessing_forkserver

18 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_multiprocessing_forkserver

Total duration: 27 min 13 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5219, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

vstinner added a commit that referenced this pull request Jun 18, 2020
@vstinner
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (G…
e805618 
…H-20944) (GH-20962) (GH-20966)

On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.

(cherry picked from commit ddbeb2f)
(cherry picked from commit b1e7361)
@bedevere-bot

bedevere-bot commented Jun 18, 2020

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.8 has failed when building commit e805618.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/273/builds/269) and take a look at the build logs.
  4. Check if the failure is related to this commit (e805618) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/273/builds/269

Failed tests:

  • test_multiprocessing_forkserver

Failed subtests:

  • test_resource_tracker_reused - test.test_multiprocessing_forkserver.TestResourceTracker

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

400 tests OK.

10 slowest tests:

  • test_concurrent_futures: 3 min 13 sec
  • test_multiprocessing_spawn: 3 min 8 sec
  • test_tools: 2 min 19 sec
  • test_tokenize: 2 min 19 sec
  • test_multiprocessing_forkserver: 2 min
  • test_lib2to3: 1 min 35 sec
  • test_asyncio: 1 min 23 sec
  • test_io: 55.4 sec
  • test_logging: 50.4 sec
  • test_pickle: 49.7 sec

1 test failed:
test_multiprocessing_forkserver

22 tests skipped:
test_devpoll test_epoll test_gdb test_idle test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tcl test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_winconsoleio test_winreg
test_winsound test_zipfile64

1 re-run test:
test_multiprocessing_forkserver

Total duration: 22 min 6 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.8.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5180, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@1st1 1st1 Awaiting requested review from 1st1
@asvetlov asvetlov Awaiting requested review from asvetlov
@vsajip vsajip Awaiting requested review from vsajip

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vstinner @mcepl @bedevere-bot @the-knights-who-say-ni