Weird test errors on OSX

@tacaswell

Last successful test run on OSX was: https://travis-ci.org/MacPython/matplotlib-wheels/builds/158635805, 4 months ago.

Errors started 24 days ago, at https://travis-ci.org/MacPython/matplotlib-wheels/builds/181269028:

======================================================================
ERROR: Failure: RuntimeError ('list' must be None or a list, not <class 'str'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/nose/loader.py", line 285, in generate
    for test in g():
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/matplotlib/testing/decorators.py", line 216, in test
    baseline_dir, result_dir = _image_directories(self._func)
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/matplotlib/testing/decorators.py", line 399, in _image_directories
    mod_file = find_dotted_module(func.__module__)[1]
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/matplotlib/testing/decorators.py", line 389, in find_dotted_module
    res = file, path, _ = imp.find_module(sub_mod, path)
  File "/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/imp.py", line 270, in find_module
    "not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>

https://travis-ci.org/MacPython/matplotlib-wheels/jobs/181269039

Errors on Python 3.4 and Python 3.5, but not 2.7.

Investigation by TC at the time at : https://gitter.im/matplotlib/matplotlib?at=58457e49b4ffd59e38ea84c8, further pointing to http://bugs.python.org/issue26844 , but this is a patch to fix the error message, to make it more informative.

I couldn't reproduce the error on my local desktop, running OSX 10.11.

Thoughts:

travis recently replaced its OSX default runtime; need to replicate OSX / xcode version for last successful build, to see if it could be an infrastructure change;
investigate dependencies. TC mentioned namespace packages - @tacaswell - was that from some further investigation you did?

Attempt to replicate previous successful build also fails : https://travis-ci.org/MacPython/matplotlib-wheels/builds/187466773 - with same errors as more recent builds.

Worker is : instance: 764e7791-10cc-4cea-a514-44971a427b99:travis-ci-osx10.9-xcode6.1-1472151920

This compared to successful build : instance: 195a0977-591b-4bc7-93a0-c1a164f57944:travis-osx10.9-sauce-xcode61-20150528

So - not exactly the same worker. At least this excludes changes in matplotlib code.

Can't replicate failure with latest Python 3.5 on OSX 10.11 on desktop. Pip freeze output:

cycler==0.10.0
matplotlib==0+unknown
mock==2.0.0
nose==1.3.7
numpy==1.11.3
pbr==1.10.0
pyparsing==2.1.10
python-dateutil==2.6.0
pytz==2016.10
six==1.10.0

Trying to replicate the dependencies in the failed travis run doesn't replicate the error:

alabaster==0.7.9
Babel==2.3.4
cycler==0.10.0
docutils==0.13.1
imagesize==0.7.1
Jinja2==2.8.1
MarkupSafe==0.23
matplotlib==0+unknown
mock==2.0.0
nose==1.3.7
numpy==1.11.0
pbr==1.10.0
Pygments==2.1.3
pyparsing==2.1.10
python-dateutil==2.6.0
pytz==2016.10
requests==2.12.4
six==1.10.0
snowballstemmer==1.2.1
Sphinx==1.5.1

Errors occurring for:

mpl_toolkits.tests.test_mplot3d
mpl_toolkits.tests.test_axes_grid1
mpl_toolkits.tests.test_axes_grid

although though these pass in the midst of all the errors:

mpl_toolkits.tests.test_mplot3d.test_wireframe3dzerostrideraises ... ok
...
mpl_toolkits.tests.test_axes_grid1.test_axesgrid_colorbar_log_smoketest

https://api.travis-ci.org/jobs/187483606/log.txt?deansi=true

@tacaswell

Moving the debug print statement finds the problem:

Trying import mpl_toolkits path None
Trying import tests path _NamespacePath(['/Users/travis/build/MacPython/matplotlib-wheels/venv/lib/python3.5/site-packages/mpl_toolkits'])
Failure: RuntimeError ('list' must be None or a list, not <class 'str'>) ... ERROR

https://api.travis-ci.org/jobs/187501264/log.txt?deansi=true

I guess what is happening is that import mpl_toolkits is generating an ImportError, triggering the except ImportError clause in find_dotted:

        def find_dotted_module(module_name, path=None):
            """A version of imp which can handle dots in the module name.
               As for imp.find_module(), the return value is a 3-element
               tuple (file, pathname, description)."""
            res = None
            for sub_mod in module_name.split('.'):
                try:
                    print('Trying import', sub_mod, 'path', path)
                    res = file, path, _ = imp.find_module(sub_mod, path)
                    path = [path]
                    if file is not None:
                        file.close()
                except ImportError:
                    # assume namespace package
                    path = sys.modules[sub_mod].__path__
                    res = None, path, None
            return res

This sets path to be a _NamespacePath object, which causes an error in the next iteration of the loop, because path must be a list or None.

@tacaswell - I don't understand the namespace stuff - any insight here? I can't explain the difference between the travis-ci machine and my local machine - I guess it could be something to do with setuptools - or anything else. Can you suggest anything to try to debug further?

I also do not understand the namespace stuff. I think this is a bug in what ever is setting __path__. Tracking through the python docs:

https://docs.python.org/3/reference/import.html#module-path says that __path__ must be an iterable of strings and that it must follow the
https://docs.python.org/3/library/sys.html#sys.path says it must be a list of strings

The explicit test on isinstance(path, (type(None), list)) was added in 2012 when find_module was re-implemented (python/cpython@b1772e6).

The source of the _NamespacePath comes from python/cpython@8c71522 and was last touched in may of last year which makes me think that this may be due to a micro-version change in python.

Relevant upstream bugs look like https://bugs.python.org/issue24192 and https://bugs.python.org/issue23911

BF: Convert namespace path to list

How confusing - I am using the latest latest Python.org Python on my desktop.

Anyway - this seems to fix it - is it the right patch?

matplotlib/matplotlib#7703

Run showing more ordinary test failures with this patch applied:

https://travis-ci.org/MacPython/matplotlib-wheels/builds/187519568

Well - that was an ugly few hours of debugging ... at least a good rest from the code of conduct discussion.

This should probably get escalated to core python? Not sure I understand it well enough to make a sensible bug report....

Oh dear - I don't understand it either. Anyone on the team better informed on namespaces that can advise me?

Navigation Menu

Repository navigation

Weird test errors on OSX #2

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Weird test errors on OSX #2

Description

Activity

matthew-brett commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

tacaswell commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

tacaswell commented on Dec 29, 2016

matthew-brett commented on Dec 29, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions