You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One bottleneck for development is a lack of ability for parallel compilation - which is something that's true for all extension code (I think this is due to a limitation in distutils / setuptools / etc). I know I'm not alone in having tried to do this before, but I just came across this SO post suggesting a way to monkeypatch distutils into submission for this purpose, so perhaps someone's feeling optimistic and wants to give this another stab?
Its a good idea. Personally, I tend to build in place (doesn't work for python3) and therefore I seem to get away with only re-building code that has changed, so for me the biggest benefit of this would be in the initial build from source.
We already parallelize the 2to3 work (which takes significantly longer than any of the C++ compilation), so I don't see why we couldn't do this as well. Since I plan to overhaul the whole setup.py script in #1454, we probably would want to base this work on that -- or wait for that to be merged.
BTW -- I use ccache, which while it doesn't parallelize, makes subsequent builds much faster through caching.
But just so we're clear: Are you finding that C++ code is getting rebuilt unnecessarily? For example, if you build, change some Python code, and then build again, no C++ code should be recompiled. Are you seeing something else? I'd still consider that a bug.
@mdboom - no, python only code changes don't affect rebuilding. Switching between master and v1.2.x is what's doing it for me - and it's only a partial rebuild right now, but it still takes a while (though, with ccache, of course, it goes down significantly after the first run!).
Ok -- should we close this bug? It's technically not resolved (i.e. it would still be very nice to have parallel builds), but ccache is a reasonable solution to the "branch switching" problem...
Great. Note also that on some platforms (at least Fedora and Debian that I'm aware of), simply installing ccache is enough to make it become the default compiler.
Activity
pelson commentedon Nov 16, 2012
Its a good idea. Personally, I tend to build in place (doesn't work for python3) and therefore I seem to get away with only re-building code that has changed, so for me the biggest benefit of this would be in the initial build from source.
mdboom commentedon Nov 16, 2012
We already parallelize the 2to3 work (which takes significantly longer than any of the C++ compilation), so I don't see why we couldn't do this as well. Since I plan to overhaul the whole
setup.pyscript in #1454, we probably would want to base this work on that -- or wait for that to be merged.BTW -- I use
ccache, which while it doesn't parallelize, makes subsequent builds much faster through caching.ivanov commentedon Nov 17, 2012
holy crap!
ccachemakes a world of difference!For me, master builds in 49 seconds without
ccache, but only 2 seconds with it enabled!without
ccache, your setup_rework branch builds for me in ~57 seconds of wall time, with it (on a repeat run), it's only ~9!moral of the story - everyone should use ccache when developing matplotlib.
mdboom commentedon Nov 19, 2012
Yes.
ccacheis awesome.But just so we're clear: Are you finding that C++ code is getting rebuilt unnecessarily? For example, if you build, change some Python code, and then build again, no C++ code should be recompiled. Are you seeing something else? I'd still consider that a bug.
ivanov commentedon Nov 19, 2012
@mdboom - no, python only code changes don't affect rebuilding. Switching between
masterandv1.2.xis what's doing it for me - and it's only a partial rebuild right now, but it still takes a while (though, withccache, of course, it goes down significantly after the first run!).mdboom commentedon Nov 27, 2012
Ok -- should we close this bug? It's technically not resolved (i.e. it would still be very nice to have parallel builds), but ccache is a reasonable solution to the "branch switching" problem...
ivanov commentedon Nov 27, 2012
Thanks for the tip, sounds reasonable to me, closing.
Just to document how one can use ccache when installing matplotlib from source:
mdboom commentedon Nov 28, 2012
Great. Note also that on some platforms (at least Fedora and Debian that I'm aware of), simply installing ccache is enough to make it become the default compiler.
pelson commentedon Feb 21, 2018
Just to follow up with this ancient thread, the following works a treat for me on a mac: