Skip to content

Wheel distributables include unnecessary files #7745

New issue
New issue
Closed
Closed
Wheel distributables include unnecessary files#7745
Labels
Build
Milestone
v2.0.0

Description

@HolgerPeters
HolgerPeters
opened on Jan 5, 2017

Context

We are using matplotlib, and we are building our own wheel-distributables built for our systems. Basically, we are not only doing this for matplotlib, but for most python dependencies of our services.
We are using the devpi-builder script (https://github.com/blue-yonder/devpi-builder) to do this, which essentially wraps pip wheel PACKAGE calls.

Issue

The issue with matplotlib is, that it is set up in a way, that test images end up in the resulting wheels, this leads to wheel sizes of up to 50 MBs, when without these files and directories, wheel sizes would be ~ 6-13 MB.

Note that on Pypi, matplotlib wheels are smaller, i.e. they don't contain these files and directories.

I looked around a little and stumbled over https://github.com/MacPython/matplotlib-wheels/blob/master/rm_test_images.py . I.e. We are not the only ones facing this problem and I assume that somewhere in the manylinux wheel build scripts there probably is a similar logic to remove these files. I'd rather have it the other way around: Don't have these files included by default.

How it should be

Matplotlib's default behaviour should be to not include the test images by default. If there are use cases where such files are needed in a wheel, inclusion should only be opt-in.

So, pip wheel matplotlib shouldn't produce a wheel that contain the large test images.

PS

Actually I'd prefer wheel releases to be free of any test-suite related stuff. If I need to run unit tests against a built Python wheel, I can take the test suite from the source distribution (as in http://doc.devpi.net/latest/quickstart-releaseprocess.html#devpi-test-testing-an-uploaded-package) if need be. Anyway, for the scope of this issue, I'd be happy if wheel sizes were reasonably small by default.

Activity

  1. tacaswell
    added this to the milestone on Jan 5, 2017
  2. tacaswell
    added
    Build
    on Jan 5, 2017

  3. tacaswell commented on Jan 5, 2017

    @tacaswell
    tacaswell
    on Jan 5, 2017
    Member

    Any idea how to change this behavior? I would be happy to merge a PR making this change.

  4. dopplershift commented on Jan 5, 2017

    @dopplershift
    dopplershift
    on Jan 5, 2017
    Contributor

    👍 from me too. Given that the official wheels are smaller, it would seem to me to be a matter of changing the defaults? What part of the release process would keep the images out of the wheels?

  5. StephanErb commented on Jan 6, 2017

    @StephanErb
    StephanErb
    on Jan 6, 2017
    Contributor

    The build system seems to have support for this already and we just have to change some defaults.

    On a clean matplotlib checkout:

    $ python setup.py sdist bdist_wheel
$ ls -lh dist/
total 65M
-rw-r--r-- 1 vagrant vagrant 32M Jan  6 18:59 matplotlib-2.0.0rc2+2914.g1fa4dd7-cp27-cp27mu-linux_x86_64.whl
-rw-r--r-- 1 vagrant vagrant 33M Jan  6 18:58 matplotlib-2.0.0rc2+2914.g1fa4dd7.tar.gz

    If I create a setup.cfg:

    [packages]
tests = False
toolkit_tests = False
sample_data = False

    and then run the build procedure again on a clean state, we see that the sdist is still large whereas the size of the wheel has reduced dramatically.

    $ rm -rf dist build
$ python setup.py sdist bdist_wheel
$ ls -lh dist/
total 41M
-rw-r--r-- 1 vagrant vagrant 8.4M Jan  6 19:02 matplotlib-2.0.0rc2+2914.g1fa4dd7-cp27-cp27mu-linux_x86_64.whl
-rw-r--r-- 1 vagrant vagrant  33M Jan  6 19:02 matplotlib-2.0.0rc2+2914.g1fa4dd7.tar.gz

    We can also simulate the usecase of creating a wheel from a pre-existing sdist. Again, the wheel size is significantly smaller than the source distribution:

    $ pip wheel dist/matplotlib-2.0.0rc2+2914.g1fa4dd7.tar.gz
$ ls -lh mat*.whl
-rw-r--r-- 1 vagrant vagrant 8.4M Jan  6 19:11 matplotlib-2.0.0rc2+2914.g1fa4dd7-cp27-cp27mu-linux_x86_64.whl

    I would therefore propose that we add a default setup.cfg to the repository with the content above. This would also allow us to simplify the conda build recipe. (Note: I have not done any further inspection besides looking at the package sizes.)

    What do yo think?

  6. tacaswell commented on Jan 6, 2017

    @tacaswell
    tacaswell
    on Jan 6, 2017
    Member

    It is probably better to find where we use those values and flip the default to False rather than to have a default setup.cfg (I think some of the other down stream packagers (mostly linux distros) prefer we not have a setup.cfg file in the source).

  7. StephanErb commented on Jan 6, 2017

    @StephanErb
    StephanErb
    on Jan 6, 2017
    Contributor

    I have created pull request #7757 to fix this issue and followed your suggestion to change the default.

  8. tacaswell
    modified the milestones: , on Jan 7, 2017
  9. QuLogic
    closed this as completedon Jan 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Build

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions