Skip to content

bpo-33735: Fix test_multiprocessing random failure#8059

Merged
vstinner merged 1 commit into
python:masterfrom
vstinner:mp_imap_unordered
Jul 3, 2018
Merged

bpo-33735: Fix test_multiprocessing random failure#8059
vstinner merged 1 commit into
python:masterfrom
vstinner:mp_imap_unordered

Conversation

@vstinner

@vstinner vstinner commented Jul 3, 2018 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

When hunting for memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

  • Only test 10 numbers instead of 100: it's a pool of 4 processes, so
    10 is enough to test at least one number per process
  • Use chunksize=100 instead of chunksize=53 to mimick test_imap()

https://bugs.python.org/issue33735

@vstinner
bpo-33735: Fix test_multiprocessing random failure
8e366e6 
When hunting for memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 100: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
@vstinner

vstinner commented Jul 3, 2018

Copy link
Copy Markdown
Member Author

I validated on my laptop that the change fixes the false alarm.

I modified locally regrtest to make it stricter:

diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 6724488fcf..a3c50e21e0 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -101,7 +101,7 @@ def dash_R(the_module, test, indirect_test, huntrleaks):
     failed = False
     for deltas, item_name, checker in [
         (rc_deltas, 'references', check_rc_deltas),
-        (alloc_deltas, 'memory blocks', check_rc_deltas),
+        (alloc_deltas, 'memory blocks', check_fd_deltas),
         (fd_deltas, 'file descriptors', check_fd_deltas)
     ]:
         # ignore warmup runs

On master, I can easily reproduce the issue on my idle laptop:

$ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:20
Run tests sequentially
0:00:00 load avg: 0.78 [1/1] test_multiprocessing_spawn
beginning 23 repetitions
12345678901234567890123
.......................
test_multiprocessing_spawn leaked [5, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=8
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 11 sec 112 ms
Tests result: FAILURE

With the change, the bug goes away:

vstinner@apu$ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:20
Run tests sequentially
0:00:00 load avg: 0.92 [1/1] test_multiprocessing_spawn
beginning 23 repetitions
12345678901234567890123
.......................
test_multiprocessing_spawn leaked [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=1
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 9 sec 967 ms
Tests result: FAILURE

The Refleak buildbots use -R 3:3, so only the first 3 numbers matter. Without the change, the first 3 numbers are non-zero, whereas they only one is non-zero with the change.

@vstinner

vstinner commented Jul 3, 2018 •
edited
Loading

Copy link
Copy Markdown
Member Author

I also validated on the Gentoo Refleak that the change fixes the false alarm.

master branch: 2 failures on 5 runs.

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 0.41 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
test_multiprocessing_spawn leaked [2, 2, 1] memory blocks, sum=5
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 6 sec 539 ms
Tests result: FAILURE

With the change: 0 failure on 5 runs.

I also ran the following command with the change:

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3 -F
Run tests sequentially
0:00:00 load avg: 2.81 [  1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
(...)
0:02:59 load avg: 5.17 [ 25] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
0:03:09 load avg: 5.45 [ 26] test_multiprocessing_spawn
beginning 6 repetitions
123456
^C
== Tests result: INTERRUPTED ==

Test suite interrupted by signal SIGINT.
0 tests omitted:


25 tests OK.

Total duration: 3 min 10 sec
Tests result: INTERRUPTED

I interrupted the test after 25 success runs.

@vstinner
vstinner merged commit 23401fb into python:master Jul 3, 2018
@miss-islington

miss-islington commented Jul 3, 2018

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7.
🐍🍒⛏🤖

@vstinner
vstinner deleted the mp_imap_unordered branch July 3, 2018 11:20
@bedevere-bot

bedevere-bot commented Jul 3, 2018

Copy link
Copy Markdown

GH-8060 is a backport of this pull request to the 3.7 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 3, 2018
@vstinner @miss-islington
bpo-33735: Fix test_multiprocessing random failure (pythonGH-8059)
ba3fec9 
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
(cherry picked from commit 23401fb)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
@miss-islington

miss-islington commented Jul 3, 2018

Copy link
Copy Markdown
Contributor

Sorry, @vstinner, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 23401fb960bb94e6ea62d2999527968d53d3fc65 2.7

@bedevere-bot

bedevere-bot commented Jul 3, 2018

Copy link
Copy Markdown

GH-8061 is a backport of this pull request to the 2.7 branch.

@bedevere-bot

bedevere-bot commented Jul 3, 2018

Copy link
Copy Markdown

GH-8062 is a backport of this pull request to the 3.6 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 3, 2018
@vstinner @miss-islington
bpo-33735: Fix test_multiprocessing random failure (pythonGH-8059)
49c2d1c 
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
(cherry picked from commit 23401fb)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
miss-islington added a commit that referenced this pull request Jul 3, 2018
@miss-islington @vstinner
bpo-33735: Fix test_multiprocessing random failure (GH-8059)
42b2f84 
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
(cherry picked from commit 23401fb)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
vstinner added a commit that referenced this pull request Jul 3, 2018
@vstinner
bpo-33735: Fix test_multiprocessing random failure (GH-8059) (GH-8061)
53fafaf 
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()

(cherry picked from commit 23401fb)
miss-islington added a commit that referenced this pull request Jul 3, 2018
@miss-islington @vstinner
bpo-33735: Fix test_multiprocessing random failure (GH-8059)
3bd9d3b 
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
  10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
(cherry picked from commit 23401fb)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vstinner vstinner

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vstinner @miss-islington @bedevere-bot @the-knights-who-say-ni