Skip to content

python setup.py build (in parallel) #1507

Closed

Description

@ivanov

One bottleneck for development is a lack of ability for parallel compilation - which is something that's true for all extension code (I think this is due to a limitation in distutils / setuptools / etc). I know I'm not alone in having tried to do this before, but I just came across this SO post suggesting a way to monkeypatch distutils into submission for this purpose, so perhaps someone's feeling optimistic and wants to give this another stab?

Activity

  1. pelson commented on Nov 16, 2012

    @pelson
    Member

    Its a good idea. Personally, I tend to build in place (doesn't work for python3) and therefore I seem to get away with only re-building code that has changed, so for me the biggest benefit of this would be in the initial build from source.

  2. mdboom commented on Nov 16, 2012

    @mdboom
    Member

    We already parallelize the 2to3 work (which takes significantly longer than any of the C++ compilation), so I don't see why we couldn't do this as well. Since I plan to overhaul the whole setup.py script in #1454, we probably would want to base this work on that -- or wait for that to be merged.

    BTW -- I use ccache, which while it doesn't parallelize, makes subsequent builds much faster through caching.

  3. ivanov commented on Nov 17, 2012

    @ivanov
    MemberAuthor

    holy crap! ccache makes a world of difference!

    For me, master builds in 49 seconds without ccache, but only 2 seconds with it enabled!

    without ccache, your setup_rework branch builds for me in ~57 seconds of wall time, with it (on a repeat run), it's only ~9!

    moral of the story - everyone should use ccache when developing matplotlib.

  4. mdboom commented on Nov 19, 2012

    @mdboom
    Member

    Yes. ccache is awesome.

    But just so we're clear: Are you finding that C++ code is getting rebuilt unnecessarily? For example, if you build, change some Python code, and then build again, no C++ code should be recompiled. Are you seeing something else? I'd still consider that a bug.

  5. ivanov commented on Nov 19, 2012

    @ivanov
    MemberAuthor

    @mdboom - no, python only code changes don't affect rebuilding. Switching between master and v1.2.x is what's doing it for me - and it's only a partial rebuild right now, but it still takes a while (though, with ccache, of course, it goes down significantly after the first run!).

  6. mdboom commented on Nov 27, 2012

    @mdboom
    Member

    Ok -- should we close this bug? It's technically not resolved (i.e. it would still be very nice to have parallel builds), but ccache is a reasonable solution to the "branch switching" problem...

  7. ivanov commented on Nov 27, 2012

    @ivanov
    MemberAuthor

    Thanks for the tip, sounds reasonable to me, closing.

    Just to document how one can use ccache when installing matplotlib from source:

    CC="ccache gcc" python setup.py install
    
  8. mdboom commented on Nov 28, 2012

    @mdboom
    Member

    Great. Note also that on some platforms (at least Fedora and Debian that I'm aware of), simply installing ccache is enough to make it become the default compiler.

  9. pelson commented on Feb 21, 2018

    @pelson
    Member

    Just to follow up with this ancient thread, the following works a treat for me on a mac:

     CC="ccache gcc" CXX="ccache" pip install -e .
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions