Cannot run setup.py build with numpy master

Since numpy/numpy#7853 landed in master, running python setup.py build for matplotlib will now fail during the build_ext section:

running build_ext
Traceback (most recent call last):
  File "setup.py", line 304, in <module>
    **extra_args
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 147, in run
    return BuildExtCommand.run(self)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/build_ext.py", line 66, in run
    _build_ext.run(self)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 164, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python2.7/distutils/command/build_ext.py", line 337, in run
    self.build_extensions()
  File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 242, in cython_sources
    extension.include_dirs = list(extension.include_dirs)
  File "~/matplotlib/setupext.py", line 910, in __get__
    result = obj._hooks[self._name]() + result
  File "~/matplotlib/setupext.py", line 937, in include_dirs_hook
    reload(numpy)
  File "/usr/local/lib/python2.7/dist-packages/numpy-1.12.0.dev0+5130ef1-py2.7-linux-x86_64.egg/numpy/__init__.py", line 117, in <module>
    raise RuntimeError('Reloading numpy is not supported')
RuntimeError: Reloading numpy is not supported

It looks like the reload was introduced by 373c4d7, but I'm not sure why it's needed or if it can be worked around.

oy... yeah I know why that is there. This is going to get hairy to fix.

The situation we were trying to fix for here was the whole problem where numpy is a dependency of the setup/build/install process, but it isn't installed ahead of time. So we used a trick to add numpy to the setup_requires entry of setup(), which would download and locally install numpy if numpy isn't already installed in order to have something to build against.

Now, I am trying to remember exactly why the reload is needed, but I remember the whole thing being very fragile in the first place. I am going to need to search my brain for a moment.

@njsmith

373c4d7 looks pretty old. Perhaps the pypi problems it was intended to solve have been fixed. @njsmith Thoughts?

@mdboom

Looking back over that original PR, I see that part of the trick was to implement a sort of "DelayedExtension" class. I am wondering if that might be the reason for the reload? @mdboom might be able to remember.

I should note that it wasn't PyPi that we were trying to deal with, it was the whole "needing numpy for build prior to installing" problem. This was our clever solution that we implemented 3 years ago.

Why do you delete __NUMPY_SETUP__? That is the only thing I can see that requires a reload after import.

That's a pretty complicated setup script:) The whole DelayedExtension class seems unnecessary, nowadays just relying on pip to deal with setup_requires and install_requires should work for getting numpy.get_include() available before starting to build your own extensions.

Also distribute_setup.py is outdated and is better to remove completely.

@rgommers

@rgommers any pointers on docs of how to do this?

It looks like we need the includes to construct the Extension objects to pass into setup() via ext_modules, thus we need to have numpy installed before setup has a chance to call pip to install numpy.

@rgommers

I think the pandas setup.py might have an appropriate incantation.

On Aug 22, 2016 14:47, "Thomas A Caswell" notifications@github.com wrote:

@rgommers https://github.com/rgommers any pointers on docs of how to do
this?

It looks like we need the includes to construct the Extension objects to
pass into setup() via ext_modules, thus we need to have numpy installed
before setup has a chance to call pip to install numpy.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6928 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAlOaGV3Z35sfCwAhgkhvPFeRiYV9873ks5qihkCgaJpZM4JfW8O
.

pandas seems to do this via numpy_incl = pkg_resources.resource_filename('numpy', 'core/include') and extensive use of the cmdclass machinery.

skimage seems to mostly use from numpy.distutils.misc_util import Configuration, get_numpy_include_dirs and a recursive (?!) setup with setup.py files in all of their sub-packages

scipy seems do do the same recursive setup.py scheme as skimage, but uses np.get_include().

sklearn looks like it has a hard numpy dependency in the setup.py

The key in pandas seems to be

from distutils.command.build_ext import build_ext as _build_ext
class build_ext(_build_ext):
    def build_extensions(self):
        numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')

        for ext in self.extensions:
            if hasattr(ext, 'include_dirs') and not numpy_incl in ext.include_dirs:
                ext.include_dirs.append(numpy_incl)
        _build_ext.build_extensions(self)

which tacks the numpy onto the include_dirs of all of the extensions

and then a

cmd_class['build_ext'] = build_ext

(pandas actually uses a further sub-class of build_ext to but details)

This (presumably) gets run late enough numpy is installed and by using pkg_resource it side-steps some of the 'fully-installed' issues?

any pointers on docs of how to do this?

There are no good docs I'm afraid, just looking at how other packages do this.

The pandas use of pkg_resources.resource_filename('numpy', 'core/include') assumes things about the internal layout of numpy's files, so don't use that. np.get_include() is the public function to use.

I think for large packages the recursive setup.py is much more maintainable than everything in one massive pile, so the way to go. But if you don't want that, it's possible to keep it together. The key thing though is to split code that deals with your external dependencies from code needed to build MPL itself. That way you should always be able to just do np.get_include for your own extensions, without needing complicated distutils command classes or things like DelayedExtension .

I suspect that the path of least code-deformation here is to use the pandas build_ext class instead of our deferred solution.

It is not clear to me if what we currently have, modulo the numpy reload issues, works with just distutils (and not setuptools).

At any rate, this will make it in for 2.0, I am just hoping someone does in before I get to it 😉

"just distutils" is irretrievably broken in general (unfortunately), e.g. you can't properly uninstall packages installed with plain distutils. So requiring setuptools is pretty reasonable these days.

Indeed. Plus that it messes up dev version strings (messes up PEP 440), which is the issue that pushed us over the line for numpy and scipy to start having a hard dependency on setuptools.

numpy has 'fixed' this upstream numpy/numpy#7955

We probably should still fix our setup.py, but it is less urgent

At some point, we will probably have to rework our build system, but considering numpy "fixed" this problem for us, I'll bump this to 2.1.

Navigation Menu

Repository navigation

Cannot run `setup.py build` with numpy master #6928

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot run setup.py build with numpy master #6928

Description

Activity

WeatherGod commented on Aug 8, 2016

charris commented on Aug 8, 2016

WeatherGod commented on Aug 8, 2016

WeatherGod commented on Aug 8, 2016

charris commented on Aug 10, 2016

rgommers commented on Aug 21, 2016

tacaswell commented on Aug 22, 2016

njsmith commented on Aug 23, 2016

tacaswell commented on Aug 23, 2016

rgommers commented on Sep 3, 2016

tacaswell commented on Sep 6, 2016

njsmith commented on Sep 6, 2016

rgommers commented on Sep 6, 2016

tacaswell commented on Sep 9, 2016

NelleV commented on Nov 10, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Cannot run `setup.py build` with numpy master #6928