Unicode Encode Error when writing pandas df to csv

Question

I cleaned 400 excel files and read them into python using pandas and appended all the raw data into one big df.

Then when I try to export it to a csv:

df.to_csv("path",header=True,index=False)

I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xc7' in position 20: ordinal not in range(128)

Can someone suggest a way to fix this and what it means?

Thanks

Please tell if you're using python 2 or 3, those two handle Unicode very differently. Then read the Unicode howto, it gives a good background for this error. These problems are not difficult to solve, but you have to know the basics first. — roeland
– roeland, Commented Jul 10, 2015 at 2:13

unutbu · Accepted Answer · 2015-07-10 02:23:38Z

70

You have unicode values in your DataFrame. Files store bytes, which means all unicode have to be encoded into bytes before they can be stored in a file. You have to specify an encoding, such as utf-8. For example,

df.to_csv('path', header=True, index=False, encoding='utf-8')

If you don't specify an encoding, then the encoding used by df.to_csv defaults to ascii in Python2, or utf-8 in Python3.

answered Jul 10, 2015 at 2:23

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jia Gao Over a year ago

does it still matter in Python3 to explicitly add the argument 'encoding='utf-8'' as it is already the default option?

unutbu Over a year ago

@JasonGoal: It's not necessary if your code is only meant to be run with Python3, but explicitness here would allow (at least this line of) your code to run under both Python2 and Python3.

tangfucius · Accepted Answer · 2016-03-17 06:12:34Z

19

Adding an answer to help myself google it later:

One trick that helped me is to encode a problematic series first, then decode it back to utf-8. Like:

df['crumbs'] = df['crumbs'].map(lambda x: x.encode('unicode-escape').decode('utf-8'))

This would get the dataframe to print correctly too.

answered Mar 17, 2016 at 6:12

tangfucius

4447 silver badges14 bronze badges

1 Comment

RufusVS Over a year ago

I like the idea of putting the helpful tips into an answer for later googling. Stack Overflow is at least a twice daily visit during my workday.

Mohsen Rajabi Golmehr · Accepted Answer · 2024-07-19 16:19:57Z

0

Use "utf_8_sig"

    df.to_csv('df.csv', encoding='utf_8_sig')

answered Jul 19, 2024 at 16:19

Mohsen Rajabi Golmehr

493 bronze badges

1 Comment

navafolk Over a year ago

have to log in to say thank you to you, it works for me.

Collectives™ on Stack Overflow

Unicode Encode Error when writing pandas df to csv

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related