Profiling multithreaded functions doesn't work #69
Description
Hi,
I was trying to profile a function that executes a subprocess.check_call() which in turns calls a multithreaded program, which is the actual program I want to profile.
I am profiling with memory_usage, and it is always returning the same value, which is around 9MB, and I guess that is the memory used by the function that create the threads, but not by the threads all together.
The real function that I'm trying to benchmark would be this one. It is quite heavy to test, so I've written an ugly small script to reproduce the scenario. Here it is:
import sys
from multiprocessing import Pool
from memory_profiler import memory_usage
def test(n):
l = [i for i in range(n)]
def test_multip(n, np):
p = Pool(processes=np)
results = p.map(test, [n]*np)
if __name__=="__main__":
t = str(sys.argv[1])
n = int(sys.argv[2])
if t == "test":
print memory_usage((test, (n, )), max_usage=True, include_children=True)
else:
np = int(sys.argv[3])
print memory_usage((test_multip, (n, np, )), max_usage=True, include_children=True)If you benchmark directly the test function, memory_usage behaves as expected:
(master)guilc@milou-b:~/tests$ python multithread.py test 10000
[9.421875]
(master)guilc@milou-b:~/tests$ python multithread.py test 100000
[12.765625]
(master)guilc@milou-b:~/tests$ python multithread.py test 1000000
[47.67578125]
(master)guilc@milou-b:~/tests$ python multithread.py test 10000000
[396.0625]
(master)guilc@milou-b:~/tests$ python multithread.py test 100000000
[3879.73828125]
However, if you test the multithreaded version, this is what happens:
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000 1
[9.5234375]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 100000 1
[9.51953125]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 1000000 1
[9.4296875]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 1
[9.31640625]
And the same if you try with more than one thread:
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 1
[9.31640625]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 2
[9.3203125]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 3
[9.3203125]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 4
[9.3203125]
(master)guilc@milou-b:~/tests$ python multithread.py test_multip 10000000 5
[9.328125]
I hope that this is enough to understand and reproduce the problem. If you have any suggestion/idea of how to fix this, I can help on that.
Thanks a lot!