Issue with non-ascii paths in font look up

@tacaswell

I posted in #3487 and following the suggestion of @tacaswell , I create a separate issue, for easier bug tracking:

Since upgrading to 1.4.0, I get the same exception.
I'm on Mac, Mavericks, latest version. anaconda bundle.

I'm not even sure it's about matplotlib, but it crashes when it gets to font_manager.

This is the relevant output for the exception:

---> 18 import matplotlib.pyplot as plt
     19
     20 from copy import deepcopy

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py in <module>()
     25
     26 import matplotlib
---> 27 import matplotlib.colorbar
     28 from matplotlib import style
     29 from matplotlib import _pylab_helpers, interactive

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/colorbar.py in <module>()
     32 import matplotlib.artist as martist
     33 import matplotlib.cbook as cbook
---> 34 import matplotlib.collections as collections
     35 import matplotlib.colors as colors
     36 import matplotlib.contour as contour

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/collections.py in <module>()
     25 import matplotlib.artist as artist
     26 from matplotlib.artist import allow_rasterization
---> 27 import matplotlib.backend_bases as backend_bases
     28 import matplotlib.path as mpath
     29 from matplotlib import _path

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/backend_bases.py in <module>()
     54
     55 import matplotlib.tight_bbox as tight_bbox
---> 56 import matplotlib.textpath as textpath
     57 from matplotlib.path import Path
     58 from matplotlib.cbook import mplDeprecation

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/textpath.py in <module>()
     17 from matplotlib.path import Path
     18 from matplotlib import rcParams
---> 19 import matplotlib.font_manager as font_manager
     20 from matplotlib.ft2font import FT2Font, KERNING_DEFAULT, LOAD_NO_HINTING
     21 from matplotlib.ft2font import LOAD_TARGET_LIGHT

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py in <module>()
   1410                 verbose.report("Using fontManager instance from %s" % _fmcache)
   1411         except:
-> 1412             _rebuild()
   1413     else:
   1414         _rebuild()

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py in _rebuild()
   1395     def _rebuild():
   1396         global fontManager
-> 1397         fontManager = FontManager()
   1398         if _fmcache:
   1399             pickle_dump(fontManager, _fmcache)

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py in __init__(self, size, weight)
   1035         #  Load TrueType fonts and create font dictionary.
   1036
-> 1037         self.ttffiles = findSystemFonts(paths) + findSystemFonts()
   1038         self.defaultFamily = {
   1039             'ttf': 'Bitstream Vera Sans',

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py in findSystemFonts(fontpaths, fontext)
    320                     fontfiles[f] = 1
    321
--> 322             for f in get_fontconfig_fonts(fontext):
    323                 fontfiles[f] = 1
    324

/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py in get_fontconfig_fonts(fontext)
    272         pipe = subprocess.Popen(['fc-list', '--format=%{file}\\n'],
    273                                 stdout=subprocess.PIPE,
--> 274                                 stderr=subprocess.PIPE)
    275         output = pipe.communicate()[0]
    276     except (OSError, IOError):

/Users/ifmichael/anaconda/lib/python2.7/subprocess.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
    708                                 p2cread, p2cwrite,
    709                                 c2pread, c2pwrite,
--> 710                                 errread, errwrite)
    711         except Exception:
    712             # Preserve original exception in case os.close raises.

/Users/ifmichael/anaconda/lib/python2.7/subprocess.pyc in _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
   1325                         raise
   1326                 child_exception = pickle.loads(data)
-> 1327                 raise child_exception
   1328
   1329

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Right now I downgraded to 1.3.1, and everything got back to normal.

Any chance you could get us a list of what files these are?

L1038 in font_manager.py:

self.ttffiles = findSystemFonts(paths) + findSystemFonts()

For reference the character it is barfing on is 'â'

Related to #3516

I've never been able to figure out what encoding fc-list is supposed to put out. Filesystem encoding, I would guess, but it's only a guess. We could try decoding the output using getfilesystemencoding()...

These are the last three time this got touched, I am stumped....

0b604b3

8e52475

8c200da

Ah -- we're already using getfilesystemencoding. I guess that's not right. Not sure what is, though.

@tacaswell

@tacaswell
Were you referring to me when asking for the list of fonts?
I don't know what am I supposed to do.

@wernerfb

@wernerfb can you take a look at this one too?

@ifmihai

@ifmihai Run the command fc-list --format=%{file}\\n from the command line.

cc @mdehoon

@wernerfb

@wernerfb Sorry, I was confused I thought these were both windows issues (some how I missed the bright red OSX label....)

I would create an issue on the IPython tracker then. Make sure to reference both this issue and your other issue with the defaultencoding over there.

Before filing a bug to ipython,
I wanted to play around more,
and I found that the issue appears in python interpreter, and spyder too.
It was a false alarm, sorry :)

The exception appears again when importing matplotlib.pyplot

I don't know if this is redundant, but i paste the error again:

2 Inn: import matplotlib.pyplot as plt
Traceback (most recent call last):

  File "<ipython-input-2-eff513f636fd>", line 1, in <module>
    import matplotlib.pyplot as plt

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py", line 27, in <module>
    import matplotlib.colorbar

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 56, in <module>
    import matplotlib.textpath as textpath

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/textpath.py", line 19, in <module>
    import matplotlib.font_manager as font_manager

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1412, in <module>
    _rebuild()

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1397, in _rebuild
    fontManager = FontManager()

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1037, in __init__
    self.ttffiles = findSystemFonts(paths) + findSystemFonts()

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py", line 322, in findSystemFonts
    for f in get_fontconfig_fonts(fontext):

  File "/Users/ifmichael/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py", line 274, in get_fontconfig_fonts
    stderr=subprocess.PIPE)

  File "/Users/ifmichael/anaconda/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)

  File "/Users/ifmichael/anaconda/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

@ifmihai

@ifmihai Based on the tracebacks you get with matplotlib, and with trying to run the subprocess code snippet independently, I suspect the problem is related to finding the fc-list executable. Apparently it is not found at all when you try to run the snippet. When subprocess searches for it from inside font_manager, it chokes on something like the problem that @cimarronm describes above.

From a terminal, what do you get with

which fc-list
printenv PATH

this is the output:

which fc-list
/usr/local/bin/fc-list

printenv PATH
“/Users/ifmichael/anaconda/bin:/Users/ifmichael/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/ifmichael/Lucru/python/lib/scripts:/usr/local/sbin”

I saw that the issue must be in subprocess or underneath it.
That's why, after I posted here, I created an issue at anaconda project, because it's their python after all.
I don't know if they customise python.

I have the same issue on linux (centos) with anaconda. The strange thing is that I have the issue on one account but not on another.

They both use anaconda, fc-list --format="%{file}\\n" gives exactly the same result. I tried to update matplotlib to 1.5, still the same error.

Was this problem solved? I am having the same problem in ipython 4.1.2 with python version 2.7.11 on Mac OsX El Capitan.

I'm getting the same error on MacOS Sierra with Python 2.7.12, Matplotlib 2.0.0 if you do import matplotlib.pyplot, it will fail with. I tried deleting the cache as mentioned earlier.

>>> import matplotlib.pyplot
Fontconfig warning: line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf
/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py:280: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  'Matplotlib is building the font cache using fc-list. '
Found an unknown keyword in AFM header (was 'licensors,')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/usr/local/lib/python2.7/site-packages/matplotlib/colorbar.py", line 36, in <module>
    import matplotlib.contour as contour
  File "/usr/local/lib/python2.7/site-packages/matplotlib/contour.py", line 22, in <module>
    import matplotlib.font_manager as font_manager
  File "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1439, in <module>
    _rebuild()
  File "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1418, in _rebuild
    fontManager = FontManager()
  File "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1072, in __init__
    self.afmlist = createFontList(self.afmfiles, fontext='afm')
  File "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py", line 590, in createFontList
    prop = afmFontProperty(fpath, font)
  File "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py", line 504, in afmFontProperty
    if font.get_angle() != 0 or name.lower().find('italic') >= 0:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x82 in position 0: ordinal not in range(128)

Do you get this error with python3?

What is the font is causing this issue? I have a guess at how to fix it, but don't know how to test it.

FIX: always decode bytes as utf-8 in AFM headers

Python 3 is not an option.

The offending font is:

 �c�e���ۃS�V�b�N�� dfmarugothic-md-win-rksj-h

I don't even know where it came from.

Update

I removed the font and it "worked" then I ran into the "no module named externals" problem, gave up and used Time Machine to get it all back to working again.

what 'no module name externals' problem?

I suspect what happened here is that 1.5.x was installed, you imported it to get the font cache built, you installed maru-gothic (which appears to be a Japanese font), but Matplotlib did not know about it. When you updated to 2.0 the font cache got rebuilt which exposed this very old bug.

Why isn't python3 an option?

Oddly, I just remembered running into this problem the last time I installed matplotlib but I don't remember my solution. I remember very clearly inserting code into fontmanager.py to tell me what font it was failing on.

I did recently install the new version of Office for MacOS and I would guess that that, coupled along with a ill-advised "pip upgrade everything", caused the problem to be revealed.

The "no module named externals" problem is an old (closed) issue that apparently arises when you get your matplotlib libraries in an inconsistent versioning state, issue 5633.

Navigation Menu

Repository navigation

Issue with non-ascii paths in font look up #3517

29 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with non-ascii paths in font look up #3517

Description

Activity

tacaswell commented on Sep 17, 2014

mdboom commented on Sep 18, 2014

tacaswell commented on Sep 18, 2014

mdboom commented on Sep 18, 2014

ifmihai commented on Sep 19, 2014

tacaswell commented on Sep 30, 2014

tacaswell commented on Sep 30, 2014

tacaswell commented on Sep 30, 2014

29 remaining items

tacaswell commented on Dec 5, 2014

ifmihai commented on Dec 5, 2014

efiring commented on Dec 5, 2014

ifmihai commented on Dec 6, 2014

Nodd commented on Dec 4, 2015

nafizh commented on May 30, 2016

actsasgeek commented on Feb 4, 2017

tacaswell commented on Feb 4, 2017

actsasgeek commented on Feb 4, 2017

tacaswell commented on Feb 5, 2017

actsasgeek commented on Feb 5, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions