Repository navigation
Unicode path woes (on Windows?) #147
Description
@with_rw_repo('0.1.6')
def test_add_unicode(self, rw_repo):
filename = u"שלום.txt"
file_path = os.path.join(rw_repo.working_dir, filename)
open(file_path, "wb").write('something')
rw_repo.index.add([filename])
self.assert_entries([filename])
I get a UnicodeEncodeError in IndexFileSHA1Writer.write, and if I pass filename.encode('utf8') then it simply cannot find the file. I guess I could figure out what the current locale encoding is and pass the correct thing, but... thats a terrible solution, and only solves the problem when the local encoding is able to encode the filename. Also, I'm sure it would create different problems.
@with_rw_repo('0.1.6')
def test_add_unicode(self, rw_repo):
filename = u"שלום.txt"
file_path = os.path.join(rw_repo.working_dir, filename)
open(file_path, "wb").write('something')
rw_repo.git.add(rw_repo.working_dir)
rw_repo.index.commit('message')
Gives something like:
File "e:\projects\galago\gitimporter\tests.py", line 50, in add_files_to_repo
repo.index.commit(message)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\index\base.py", line 887, in commit
return Commit.create_from_tree(self.repo, tree, message, parent_commits, head)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\objects\commit.py", line 358, in create_from_tree
master = git.refs.Head.create(repo, repo.head.ref, new_commit, logmsg="commit (initial): %s" % message)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\refs\symbolic.py", line 505, in create
return cls._create(repo, path, cls._resolve_ref_on_create, reference, force, logmsg)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\refs\symbolic.py", line 472, in _create
ref.set_reference(target, logmsg)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\refs\symbolic.py", line 307, in set_reference
self.log_append(oldbinsha, logmsg)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\refs\symbolic.py", line 360, in log_append
message)
File "c:\Users\User.virtualenvs\galago\lib\site-packages\gitpython-0.3.2.rc1-py2.7.egg\git\refs\log.py", line 257, in append_entry
fd.write(repr(entry))
Windows 7, Python 2.7, GitPython 3.2.1.rc1
Activity
Byron commentedon Nov 19, 2014
I will have a look at this one when porting to py3, as it will require proper string/bytes handling.
Byron commentedon Jan 5, 2015
On OSX, I couldn't reproduce the issue.
At least the test is now part of the official test suite - will recheck on windows another time.
Something that comes to my mind is that windows might be using yet another encoding, and git-python tries to interpret it as UTF8, maybe
sys.getfilesystemencoding()should be used to do the right thing.Added test of #147 to verify it works.
Byron commentedon Jan 6, 2015
I am able to reproduce this issue on windows, here is what I get:
Apparently the encoding goes all crazy, lets see how the fix looks like.
Byron commentedon Jan 6, 2015
This is how it looks like, if I try to look at the file through a git bash. The file, however, looks good in the explorer (see bottom most file in following image)
Improved windows test suite.
Byron commentedon Jan 6, 2015
Please have a look at this altered test case for how to workaround the issue.
Interestingly, python starts to properly deal with unicode in filenames in python 3 - please have a look at the image showing how 2.7 fails, and 3.4 succeeds in the same task.
This concludes this issue, as I believe python 3 should be used if unicode is a concern, and git-pythons own facilities should be used to add unicode paths as the git comamndline tool seems to have issues with it.