UnicodeDecodeError when reading CSV file in Pandas with Python

Michael Zippo

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

I"m running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error...

File "C:Importersrcdfmanimporter.py", line 26, in import_chr
     data = pd.read_csv(filepath, names=fields)
File "C:Python33libsite-packagespandasioparsers.py", line 400, in parser_f
     return _read(filepath_or_buffer, kwds)
File "C:Python33libsite-packagespandasioparsers.py", line 205, in _read
     return parser.read()
   File "C:Python33libsite-packagespandasioparsers.py", line 608, in read
     ret = self._engine.read(nrows)
File "C:Python33libsite-packagespandasioparsers.py", line 1028, in read
     data = self._reader.read(nrows)
File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandasparser.c:6745)
File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandasparser.c:6964)
File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandasparser.c:7780)
File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandasparser.c:8793)
File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandasparser.c:9484)
File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandasparser.c:10642)
File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandasparser.c:10853)
File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandasparser.c:15657)
UnicodeDecodeError: "utf-8" codec can"t decode byte 0xda in position 6: invalid    continuation byte

The source/creation of these files all come from the same place. What"s the best way to correct this to proceed with the import?

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from UnicodeDecodeError when reading CSV file in Pandas with Python, check other code Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Walter Richtgofen

Tallinn | 2023-03-29

Maybe there are another answers? What UnicodeDecodeError when reading CSV file in Pandas with Python exactly means?. Checked yesterday, it works!

Marie Gonzalez

Warsaw | 2023-03-29

I was preparing for my coding interview, thanks for clarifying this - UnicodeDecodeError when reading CSV file in Pandas with Python in Python is not the simplest one. Will get back tomorrow with feedback

Boris Jackson

Prague | 2023-03-29

Thanks for explaining! I was stuck with UnicodeDecodeError when reading CSV file in Pandas with Python for some hours, finally got it done 🤗. Will use it in my bachelor thesis