Skip to content

Upload packages to PyPI directly, for pip 1.5 #2715

Closed

Description

@rgbkrk

As of pip 1.5 (released January 1, 2014), pip will no longer install externally hosted files by default.

If a user tries to pip install matplotlib it will default to the most recent package hosted on PyPI (1.1.1).

Pinning the dependency to install matplotlib 1.3.1 results in a helpful error message

$ pip install matplotlib==1.3.1
Downloading/unpacking matplotlib==1.3.1
  Could not find a version that satisfies the requirement matplotlib==1.3.1 (from versions: 0.86.1, 0.86.2, 0.86, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1)
  Some externally hosted files were ignored (use --allow-external to allow).
Cleaning up...
No distributions matching the version for matplotlib==1.3.1
Storing debug log for failure in /home/vagrant/.pip/pip.log

Users can work around this by either specifying to allow external and unverified,

$ pip install matplotlib==1.3.1 --allow-external matplotlib --allow-unverified matplotlib
Downloading/unpacking matplotlib==1.3.1
  matplotlib an externally hosted file and may be unreliable
  matplotlib is potentially insecure and unverifiable.
  Downloading matplotlib-1.3.1.tar.gz (42.2MB): 42.2MB downloaded
...

or by installing straight from the package for their respective system (tarball for *nix shown below)

$ pip install https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz

It would be really nice to have this package available for pip. It seems that in the past this was a filesize limit restriction workaround on PyPI, which can now handle bigger packages.

If the filesize is still an issue, perhaps we can work with the PyPI folks to either increase the limit or put matplotlib on a whitelist.

Activity

  1. mdboom commented on Jan 13, 2014

    @mdboom
    Member

    It seems matplotlib is still too big. Uploading through the web interface gives:

    Error processing form
    
    distribution file too large
    

    python setup.py sdist upload gives:

    Submitting dist/matplotlib-1.3.1.tar.gz to http://pypi.python.org/pypi
    Upload failed (400): distribution file too large
    

    Any help you can provide to make this work would be great.

  2. tacaswell commented on Jan 13, 2014

    @tacaswell
    Member

    Does the tarball strip out the testing images?

  3. dstufft commented on Jan 13, 2014

    @dstufft

    So the (obvious) two solutions are strip things out of the tarball to make it smaller, or bump up the file size. Can you tell me what the file size you tried to upload?

  4. mdboom commented on Jan 13, 2014

    @mdboom
    Member

    No, the tarball does not strip out the testing images. I'm not sure we want to do that except as a last resort.

    The source tarball is ~42MB.

  5. dstufft commented on Jan 13, 2014

    @dstufft

    Ok, that's about 2MB over the current limit. The limit is pretty arbitrary so we can bump that up a bit. I'll comment on this ticket once I get time to do that (should be sometime today).

  6. rgbkrk commented on Jan 13, 2014

    @rgbkrk
    Author

    It's primarily the lib/matplotlib/tests/baseline_images I assume that are making this so big?

    Is the purpose here so people can test their installation (or development setup)? If people use matplotlib in production settings where they repeatedly deploy it, would it make sense to make a version that doesn't bundle all this in?

  7. mdboom commented on Jan 15, 2014

    @mdboom
    Member

    If I wanted to provide packages both with and without the test data, how would you recommend doing that on pypi? Have two separate package entries, i.e. use the existing matplotlib for without tests, and matplotlib-tests for the tests (which installs into a namespace package, I guess...?)

    I don't want to change this for a bugfix release, but rather wait for our next major release (which is scheduled for "pretty soon now").

  8. mdboom commented on Jan 15, 2014

    @mdboom
    Member

    And in the interest of not changing things -- I should probably go and upload to PyPI the last few years worth of releases exactly as they were on Sourceforge -- it's actually pretty typical with matplotlib for users to specify old versions, so I'd like that to work without the --allow-unverified flags, too. This implies bumping up the file size limit, as I don't see another way around that.

  9. rgbkrk commented on Jan 21, 2014

    @rgbkrk
    Author

    Installing matplotlib-tests into a namespace package sounds good.

    I've never used it, but I imagine the syntax in setup.py would look something like:

     entry_points={
            "matplotlib": [
                "tests = matplotlib-tests"
            ]
     }

    I summon @ivanov for more expertise than myself (we discussed this in person).

  10. cbrnr commented on Feb 7, 2014

    @cbrnr
    Contributor

    I +1 the idea to bump up the file limit (e.g. 50MB should be sufficient). I just recently ran into this problem where only an old version was downloaded. I can't imagine I'm the only one trying to "pip install matplotlib", so it's definitely not good that this doesn't work (especially since I'm using Python 3, where >= 1.2.0 is required).

  11. mdboom commented on Feb 7, 2014

    @mdboom
    Member

    @dstufft: What's the status on the file size limit? I think I'd prefer that solution (for now), so as to not change our installation procedure in a significant way in a bug fix release. Is there a PyPI bug tracker entry (or some such) I could follow?

  12. cbrnr commented on Feb 7, 2014

    @cbrnr
    Contributor

    I didn't find a dedicated issue there, but I reported this problem at https://sourceforge.net/p/pypi/support-requests/342/ and pointed out the file size limitation issue.

  13. dstufft commented on Feb 7, 2014

    @dstufft

    I'm a terrible person and forgot all about this, let me do it right now.

  14. dstufft commented on Feb 7, 2014

    @dstufft

    Can you try to upload to TestPyPI and see if it works there? If it does I'll deploy it out to PyPI itself.

  15. mdboom commented on Feb 7, 2014

    @mdboom
    Member

    I get:

    Creating tar archive
    removing 'matplotlib-1.3.1' (and everything under it)
    running upload
    Submitting dist/matplotlib-1.3.1.tar.gz to http://testpypi.python.org/pypi
    error: None
    

    How would I further debug this?

  16. dstufft commented on Feb 7, 2014

    @dstufft

    o.O

    Uhhh, can you try using Twine to upload instead? https://pypi.python.org/pypi/twine/

  17. mdboom commented on Feb 7, 2014

    @mdboom
    Member

    How do I tell twine to go to testpypi?

    › twine upload -r http://testpypi.python.org/pypi dist/*                                                                                                                 1 « git · cfb6ac64578c · (unknown) »
    Traceback (most recent call last):
      File "/home/mdboom/python/bin/twine", line 11, in <module>
        sys.exit(main())
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
        return app.cli(sys.argv[1:])
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
        (k, v) for k, v in args._get_kwargs()
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 69, in __call__
        parsed = urlparse(config["repository"])
    KeyError: 'repository'
    
  18. dstufft commented on Feb 7, 2014

    @dstufft

    Make sure testpypi is in your ~/.pypirc and use -r testpypi like:

    [distutils]
    index-servers =
        pypi
        testpypi
    
    [pypi]
    username:dstufft
    password:whatever
    
    [testpypi]
    repository:https://testpypi.python.org/pypi
    username:dstufft
    password:whatever
    
  19. dstufft commented on Feb 7, 2014

    @dstufft

    FWIW i'm dstufft on freenode if you want to do this in a more immediate response format :)

  20. mdboom commented on Feb 7, 2014

    @mdboom
    Member
    Uploading distributions to https://testpypi.python.org/pypi
    Uploading matplotlib-1.3.1.tar.gz
    No handlers could be found for logger "requests.packages.urllib3.connectionpool"
    Traceback (most recent call last):
      File "/home/mdboom/python/bin/twine", line 11, in <module>
        sys.exit(main())
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
        return app.cli(sys.argv[1:])
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
        (k, v) for k, v in args._get_kwargs()
      File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 177, in __call__
        resp.raise_for_status()
      File "/home/mdboom/python/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 502 Server Error: Bad Gateway
    
  21. dstufft commented on Feb 7, 2014

    @dstufft

    Ah ha, Hmm

    Let me go dig through the logs. Did you generate it jsut with python setup.py sdist from this repo?

  22. dstufft commented on Feb 8, 2014

    @dstufft

    I believe this should be fixed now on PyPI and TestPyPI

  23. mdboom commented on Feb 8, 2014

    @mdboom
    Member

    Success (on TestPyPI).

  24. mdboom commented on Feb 8, 2014

    @mdboom
    Member

    On PyPI proper, it seems it still has the file size limit:

    requests.exceptions.HTTPError: 400 Client Error: distribution file too large
    

    Don't know if that's still to be expected. Once that's resolved, I'll upload this, and this bug can be closed...

    Thanks for all of your help.

  25. ewdurbin commented on Feb 8, 2014

    @ewdurbin

    @mdboom just verified the updated limit is in place on PyPI, would you mind trying again?

  26. mdboom commented on Feb 9, 2014

    @mdboom
    Member

    Success! Thanks for everyone's help. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions