import error when non-ascii characters are present in cwd or user name (windows)

Ahoy!
I have recently updated matplotlib through conda to version 14.0 (np19py27_0).
I am using windows 8, python 2.7.7 (Anaconda distribution 2.0.2 32-bit)
Unfortunatelly my windows user name contains non-ascii caracter (ś) and importing matplotlib gives errors. 😢 (previous matplotlib versions I used did not have any problem with this)

The error comes in two flavors:
⚫ If my current working directory contains non-ascii character then:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-82be63b7783c> in <module>()
----> 1 import matplotlib

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in <module>()
   1046 
   1047 # this is the instance used by the matplotlib classes
-> 1048 rcParams = rc_params()
   1049 
   1050 if rcParams['examples.directory']:

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in rc_params(fail_on_error)
    895     default matplotlib rc file.
    896     """
--> 897     fname = matplotlib_fname()
    898     if not os.path.exists(fname):
    899         # this should never happen, default in mpl-data should always be found

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in matplotlib_fname()
    746       system-defined copy.
    747     """
--> 748     fname = os.path.join(os.getcwd(), 'matplotlibrc')
    749     if os.path.exists(fname):
    750         return fname

D:\Python\Anaconda\lib\ntpath.pyc in join(path, *paths)
     82         if result_path and result_path[-1] not in '\\/':
     83             result_path = result_path + '\\'
---> 84         result_path = result_path + p_path
     85     return result_drive + result_path
     86 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 11: ordinal not in range(128)

⚫ If I change my current directory so that it does not contain any ascii characters the error transforms itself a little:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-6-1d1446f6fa64> in <module>()
----> 1 from matplotlib import pyplot as plt

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in <module>()
   1046 
   1047 # this is the instance used by the matplotlib classes
-> 1048 rcParams = rc_params()
   1049 
   1050 if rcParams['examples.directory']:

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in rc_params(fail_on_error)
    895     default matplotlib rc file.
    896     """
--> 897     fname = matplotlib_fname()
    898     if not os.path.exists(fname):
    899         # this should never happen, default in mpl-data should always be found

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in matplotlib_fname()
    757                 return fname
    758 
--> 759     configdir = _get_configdir()
    760     if configdir is not None:
    761         fname = os.path.join(configdir, 'matplotlibrc')

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_configdir()
    628     5. A writable directory could not be found or created; return None.
    629     """
--> 630     return _get_config_or_cache_dir(_get_xdg_config_dir())
    631 
    632 get_configdir = verbose.wrap('CONFIGDIR=%s', _get_configdir, always=False)

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_xdg_config_dir()
    553     path = os.environ.get('XDG_CONFIG_HOME')
    554     if path is None:
--> 555         path = get_home()
    556         if path is not None:
    557             path = os.path.join(path, '.config')

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in wrapper(*args, **kwargs)
    321         assert six.callable(func)
    322         def wrapper(*args, **kwargs):
--> 323             ret = func(*args, **kwargs)
    324 
    325             if (always or not wrapper._spoke):

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_home()
    507     """
    508     try:
--> 509         path = os.path.expanduser("~")
    510     except ImportError:
    511         # This happens on Google App Engine (pwd module is not present).

D:\Python\Anaconda\lib\ntpath.pyc in expanduser(path)
    271         userhome = join(dirname(userhome), path[1:i])
    272 
--> 273     return userhome + path[i:]
    274 
    275 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 13: ordinal not in range(128)

Partial duplicate of #3487.

I think this is a Python bug that is reportedly is unfixable on Python 2. See this:

http://bugs.python.org/issue13207

It looks difficult to workaround, since even if we do the home directory lookup ourselves, you can't get a unicode environment variable on Windows on Python 2. There is a proposed ctypes solution, but ick.

One possible workaround is to define the MPLCONFIGDIR environment variable to an ascii-clean path, and the configuration file for matplotlib will be stored there rather than your home directory.

@mmagnuski

Yeah, tracebacks point to the problem in the python 2 library itself.

Did previous version not check for anything in the user home path? @mmagnuski, which previous version of matplotlib worked for you in the past and were you using python2?

I changed to matplotlib 1.3.1 and importing works fine. I don't remember which version I used originally but most probably something around 1.3
But my non-unicode home path is now a problem for pip, I regret using a polish sign in my username, eh...

If the issue is the underlying python bug, why does 1.3.1 work?

Where did you install mpl on your system?

@tacaswell

@tacaswell, maybe we see the problem now because we are using unicode_literals, so we are injecting unicode where previously we were injecting native strings? In my own code porting, I concluded that using unicode_literals causes more problems than it solves. In the case of this issue, though, I am just guessing.

I tracked this back via blame and it looks like what happened is that in ca6cd19 the except was changed to only catch ImportErrors.

Older version of mpl are just silently snarfing the error.

BUG : skip looking up expanduser if unicode error

If this is relevant - I do have a .matplotlib directory present in my C:\Users\Mikuś\ path with matplotlib 1.3.1. The path has non-ascii signs but matplotlib was somehow able to create a dir there.

Can you see if this the patch in #3532 works?

The next block of code after this one tries to get the home directory from the envs for evar in ('HOME', 'USERPROFILE', 'TMP'): which might handle the unicode paths correctly.

A user of my app just run into the same problem with 1.4, downgraded my app to mpl 1.3 which does not show this problem.

@wernerfb

@wernerfb Can you check if the patch in #3532 fixes it for your user?

@wernerfb

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Hi Thomas,

I can test it here, but only on my test machine, so need to py2exe app -
which takes a bit.

Unfortunately this doesn't help, I now get the exception:
twcbsrc.app_base - ERROR - app_base.pyo - 292 - Traceback (most recent
call last):
File "twcbsrc\controllers\app_cb.pyo", line 403, in onTBStats
File "importlib__init__.pyo", line 37, in import_module
File "zipextimporter.pyo", line 82, in load_module
File "twcbsrc\controllers\app_stats.pyo", line 29, in
File "zipextimporter.pyo", line 82, in load_module
File "twcbsrc\views\statistics.pyo", line 19, in
File "zipextimporter.pyo", line 82, in load_module
File "matplotlib__init__.pyo", line 1051, in
File "matplotlib__init__.pyo", line 900, in rc_params
File "matplotlib__init__.pyo", line 762, in matplotlib_fname
File "matplotlib__init__.pyo", line 633, in get_configdir
File "matplotlib__init_.pyo", line 558, in get_xdg_config_dir
File "matplotlib__init_.pyo", line 326, in wrapper
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 32:
invalid start byte

What is strange is that this worked with 1.3 and if I do
'os.path.expanduser('~')' in the py2exe'd application using a test
Python prompt I don't get an exception either and the path shown in the
prompt is "C:\Dokumente und Einstellungen\j\xf6rg f\xfcr tests"

Could it have to do with "from future import unicode_literals" ?

Werner

@wernerfb

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Hhm, just did a little test on the Python prompt within the py2exe'd
app and yes it is the future import.

from future import unicode_literals
os.path.expanduser('~')

@wernerfb

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

looking at this thread:
https://stackoverflow.com/questions/23888120/an-alternative-to-os-path-expanduser

I tried on the test Python prompt without getting an exception and the
returned value looks correct.

from future import unicode_literals
os.path.expanduser(b'~').decode(sys.getfilesystemencoding())

@wernerfb

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Making the following change works for me on the dev machine running
Windows 8.1, Python 2.7.8, wxPython 2.9.5 and running the py2exe'd app
on Windows XP.

```
   path = os.path.expanduser("~")
```

   path = os.path.expanduser(b"~").decode(sys.getfilesystemencoding())

@wernerfb

Could you put that in a PR?

On Mon, Sep 29, 2014 at 12:03 PM, Werner F Bruhin notifications@github.com
wrote:

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
<
#3516 (comment)
.

Making the following change works for me on the dev machine running
Windows 8.1, Python 2.7.8, wxPython 2.9.5 and running the py2exe'd app
on Windows XP.

path = os.path.expanduser("~")

path = os.path.expanduser(b"~").decode(sys.getfilesystemencoding())

—
Reply to this email directly or view it on GitHub
#3516 (comment)
.

Thomas Caswell
tcaswell@gmail.com

closed by #3594

Navigation Menu

Repository navigation

import error when non-ascii characters are present in cwd or user name (windows) #3516

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

import error when non-ascii characters are present in cwd or user name (windows) #3516

Description

Activity

tacaswell commented on Sep 14, 2014

mdboom commented on Sep 16, 2014

cimarronm commented on Sep 16, 2014

mmagnuski commented on Sep 17, 2014

tacaswell commented on Sep 18, 2014

efiring commented on Sep 18, 2014

tacaswell commented on Sep 18, 2014

mmagnuski commented on Sep 18, 2014

tacaswell commented on Sep 18, 2014

wernerfb commented on Sep 29, 2014

tacaswell commented on Sep 29, 2014

wernerfb commented on Sep 29, 2014

wernerfb commented on Sep 29, 2014

wernerfb commented on Sep 29, 2014

wernerfb commented on Sep 29, 2014

tacaswell commented on Sep 29, 2014

tacaswell commented on Sep 30, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions