Repository navigation
Handling of BOM (byte order mark) in source files #4353
Description
Feature
PEP 263 specifies that
To aid with platforms such as Windows, which add Unicode BOM marks to the beginning of Unicode files, the UTF-8 signature \xef\xbb\xbf will be interpreted as ‘utf-8’ encoding as well (even if no magic encoding comment is given).
However, RustPython currently fails on files that include a BOM.
$ printf '\xEF\xBB\xBF' > f.py
$ cargo run f.py
SyntaxError: Got unexpected token at line 1 column 1
^Python Documentation
Not familiar with CPython source code, but some pointers:
https://github.com/python/cpython/blob/3.11/Lib/tokenize.py#L299 (detect_encoding)
https://github.com/python/cpython/blob/3.11/Lib/test/test_importlib/source/test_source_encoding.py
Activity
fanninpm commentedon Dec 24, 2022
I remember encountering something like this before, in #3919 (comment).
yt2b commentedon Dec 24, 2022
Checking whether BOM exists is needed.
But in Rust, UTF-8 BOM(
EF BB BF) is recognized asFE FFwhen converted from bytes to String.Update rust python to handle files with BOM
DimitrisJim commentedon Dec 26, 2022
Thanks for reporting this @squiddy! #4359 fixed the erroneous
SyntaxError, handling of the mismatch between it and an encoding declaration that differs is not handled (and silently ignored atm) so keep that in mind.