Repository navigation
Description
Bug report
Bug description:
While parsing a recent spam message, my script threw a UnicodeEncodeError exception from inside the email package. I reduced the problem to a minimal test case based on the real‑world (non‑compliant) Subject header. The header mixes raw UTF‑8 characters with RFC 2047 encoded‑words, which appears to cause the parser to produce surrogateescape code points when utf8=True is enabled.
Calling as_bytes() then triggers a failure during header folding.
I think that the package is mixing raw utf-8 handling with trying to wrap a header in =?unknown-8bit? sequences when utf-8 is true but should likely do something to treat the entirety of the header line as malformed and defensively fence it all off.
Below is a complete, self‑contained Python script that reproduces the error:
# This is a real-world spam subject line, and the testcase is distilled down
# to the form you see below with simply a subject, mime-version and short body.
#
# This script reproduces a UnicodeEncodeError in the email package
# when parsing a header containing mixed encoded-words and raw UTF-8
# under policy.default.clone(utf8=True). NB: the email is malformed
# because in mixes raw UTF-8 with encoded-words in the same header.
#
# Expected traceback:
#
expected_traceback = r"""
Traceback (most recent call last):
File "C:\Tmp\PyTest\email_package_corrupted.py", line 32, in <module>
print(msg.as_bytes())
~~~~~~~~~~~~^^
File "C:\Program Files\Python\Lib\email\message.py", line 208, in as_bytes
g.flatten(self, unixfrom=unixfrom)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 117, in flatten
self._write(msg)
~~~~~~~~~~~^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 200, in _write
self._write_headers(msg)
~~~~~~~~~~~~~~~~~~~^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 432, in _write_headers
self._fp.write(self.policy.fold_binary(h, v))
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "C:\Program Files\Python\Lib\email\policy.py", line 207, in fold_binary
folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
File "C:\Program Files\Python\Lib\email\policy.py", line 228, in _fold
return self.header_factory(name, ''.join(lines)).fold(policy=self)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\headerregistry.py", line 253, in fold
return header.fold(policy=policy)
~~~~~~~~~~~^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 166, in fold
return _refold_parse_tree(self, policy=policy)
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 2874, in _refold_parse_tree
last_ew = _fold_as_ew(tstr, lines, maxlen, last_ew,
part.ew_combine_allowed, charset, leading_whitespace)
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 3006, in _fold_as_ew
encoded_word = _ew.encode(to_encode_word, charset=encode_as)
File "C:\Program Files\Python\Lib\email\_encoded_words.py", line 222, in encode
bstring = string.encode('ascii', 'surrogateescape')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-5: ordinal not in range(128)
"""
from email import policy
from email import message_from_bytes, message_from_binary_file
# Raw email with NO Content-Type header
raw = (
b'Subject: '
b'xbox =?UTF-8?B?4puU77iP?= \xf0\x9d\x90\xa5\xf0\x9d\x90'
b'\x9a\xf0\x9d\x90\xac\xf0\x9d\x90\xad \xf0\x9d\x90\xab\xf0\x9d\x90'
b'\x9e\xf0\x9d\x90\xa6\xf0\x9d\x90\xa2\xf0\x9d\x90\xa7\xf0\x9d\x90'
b'\x9d\xf0\x9d\x90\x9e\xf0\x9d\x90\xab! \xf0\x9d\x90\x98\xf0\x9d\x90'
b'\xa8\xf0\x9d\x90\xae\xf0\x9d\x90\xab \xf0\x9d\x90\xa5\xf0\x9d\x90'
b'\xa2\xf0\x9d\x90\x9c\xf0\x9d\x90\x9e\xf0\x9d\x90\xa7\xf0\x9d\x90\xac'
b'\xf0\x9d\x90\x9e \xf0\x9d\x90\xa1\xf0\x9d\x90\x9a\xf0\x9d\x90\xac '
b'\xf0\x9d\x90\x9e\xf0\x9d\x90\xb1\xf0\x9d\x90\xa9\xf0\x9d\x90\xa2\xf0'
b'\x9d\x90\xab\xf0\x9d\x90\x9e\xf0\x9d\x90\x9d \xf0\x9d\x90\x93\xf0\x9d'
b'\x90\xa8\xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xb2 '
b'=?UTF-8?B?4pqg77iP?=... \xf0\x9d\x90\xb2\xf0\x9d\x90\xa8\xf0\x9d\x90'
b'\xae\xf0\x9d\x90\xab \xf0\x9d\x90\x9c\xf0\x9d\x90\xa8\xf0\x9d\x90\xa6'
b'\xf0\x9d\x90\xa9\xf0\x9d\x90\xae\xf0\x9d\x90\xad\xf0\x9d\x90\x9e\xf0'
b'\x9d\x90\xab \xf0\x9d\x90\xa2\xf0\x9d\x90\xac \xf0\x9d\x90\xa2\xf0\x9d'
b'\x90\xa7 \xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xa7\xf0\x9d\x90'
b'\xa0\xf0\x9d\x90\x9e\xf0\x9d\x90\xab\r\n'
b'MIME-Version: 1.0'
b'\r\n\r\n'
b'Hello, world!')
msg = message_from_bytes(raw, policy=policy.default.clone(utf8=True))
print('This call to as_bytes() should trigger the UnicodeEncodeError')
print(msg.as_bytes())
expected_traceback = r"""
This call to as_bytes() should trigger the UnicodeEncodeError
Traceback (most recent call last):
File "C:\Tmp\PyTest\email_package_corrupted.py", line 32, in <module>
print(msg.as_bytes())
~~~~~~~~~~~~^^
File "C:\Program Files\Python\Lib\email\message.py", line 208, in as_bytes
g.flatten(self, unixfrom=unixfrom)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 117, in flatten
self._write(msg)
~~~~~~~~~~~^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 200, in _write
self._write_headers(msg)
~~~~~~~~~~~~~~~~~~~^^^^^
File "C:\Program Files\Python\Lib\email\generator.py", line 432, in _write_headers
self._fp.write(self.policy.fold_binary(h, v))
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "C:\Program Files\Python\Lib\email\policy.py", line 207, in fold_binary
folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
File "C:\Program Files\Python\Lib\email\policy.py", line 228, in _fold
return self.header_factory(name, ''.join(lines)).fold(policy=self)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\headerregistry.py", line 253, in fold
return header.fold(policy=policy)
~~~~~~~~~~~^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 166, in fold
return _refold_parse_tree(self, policy=policy)
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 2874, in _refold_parse_tree
last_ew = _fold_as_ew(tstr, lines, maxlen, last_ew,
part.ew_combine_allowed, charset, leading_whitespace)
File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 3006, in _fold_as_ew
encoded_word = _ew.encode(to_encode_word, charset=encode_as)
File "C:\Program Files\Python\Lib\email\_encoded_words.py", line 222, in encode
bstring = string.encode('ascii', 'surrogateescape')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-5: ordinal not in range(128)
"""
CPython versions tested on:
3.14
Operating systems tested on:
Windows
Activity
a12k commentedon Jan 12, 2026
I can repro this. I took a look at the email package. There's a simple bandage you could apply, in the
_fold_as_ewfunction I did:To smoke test it, then convert those internal \udcxx surrogates and turn them back into the original raw bytes they were before parsing, then decode back to string. This works in the repl
While that patch works, I expect it's a little "whack-a-mole" and an actual fix is to make
email/_encoded_words.py"surrogate aware" so we prevent this class of bugs regardless of which submodule does it. But that's an exercise for whoever gets assigned to fix this. :)picnixz commentedon Jan 12, 2026
Is this allowed by RFC 6532? because you set
utf8=Trueadn the docs say:a12k commentedon Jan 12, 2026
This took longer to research than I expected, but I think yes a fix would be allowed by RFC 6532. Postel's Law aside (and despite my personal disagreement in with its principle in OSS):
Section 3.6 "Use of MIME Encoded-Words" states "Agents MAY, when incorporating material from another message, convert encoded-word use to direct use of UTF-8. (emphasis mine)" By fixing this By fixing the bug we would be performing the conversion explicitly authorized by this section.
Further, Section 3.2 "Syntax Extensions to RFC 5322" states that "constructs now allow UTF-8: 1. Unstructured text, used in header fields like 'Subject:'..." which in my reading makes the raw portions of the header in the repro script valid.
Back to Section 3.6, "...encoded-words SHOULD NOT be used when generating header fields for messages employing this extension." So the implication is that prohibition is not a mandate for cpython to crash upon receiving, but a mandate to generate utf-8.
Together, my combined read is that the repro mixing raw UTF-8 and encoded words is a "generation" error by the sender (violating the "SHOULD NOT"), but RFC 6532 Section 3.6 grants the receiver (read, cpython email package) permission to resolve this by converting the encoded parts to UTF-8. So I would advocate for handling this gracefully rather than crashing.
That being said, if fixing this is a direction we want to pursue, I did some more digging into the code and believe I could create a simple PR to handle this.
picnixz commentedon Jan 12, 2026
The question is also whether this would work with SMTP servers:
Then I read it differently. Because this would be breaking other users that correctly use this. When there is a "MAY" in an RFC, it means that we could but that's optional. It's not a bug to me, but rather a feature request which I personally don't find attractive enough. It's prone to breaking validation downstream. If we want to allow additional support, we should either create an additional policy object for that, or have a
strict=Trueby default for the policy for whichstrict=Falsewould mean "try harder".cc @bitdancer
a12k commentedon Jan 12, 2026
Yeah agree that RFC 6532 makes the conversion optional ("MAY"), but I don't think it justifies the current failure mode. My take is that we are crashing in violation of the email package's API contract. The default policy (
raise_on_defect=False) implies that malformed input should result in aemail.errors.MessageDefect, not an uncaughtUnicodeEncodeErrortraceback.By crashing, we currently force an implicit
strict=Truebehavior for this specific edge case. That prevents users from inspecting the message or handling the error gracefully.Reflecting further, my suggestion isn't that we silently fix the input, it's that we catch the encoding error, register a
MessageDefect(signaling the RFC violation), and fallback to a safe encoding (e.g., treating the segment as raw UTF-8 as permitted by Section 3.6). This preserves validation capabilities (via defects) without crashing the runtime.bitdancer commentedon Jan 12, 2026
I fixed this in #142517 so that it correctly re-encodes the unknown bytes as unknown-8bit but leaves the utf-8 alone if utf8=True. Please re-run your tests against main.
I was not aware of the RFC 6532 discouragement of encoded words when using the SMTPUTF8 extension. I may have been when I originally wrote the code, but that was a while ago and I at least didn't remember when writing the fix. That said, SHOULD NOT is not MUST NOT, which means that smtp servers must be prepared to handle encoded words in SMTPUTF8 mode. There is also the issue of dealing with signed message attachments and rfc822 attachments that shouldn't normally be modified when resent, both of which potentially require handling encoded words even if in SMPTUTF8 mode.
I think my fix is as correct as we're going to get, but I'm open to argument. The fix hasn't been released anywhere yet, so we can still change this if we decide we want to.
However this report demonstrates a deeper problem with the current support for utf8=True. On parse those utf-8 characters should have been decoded, so that the end result of refolding that header should have been all utf8. The fix is simple (checking the policy flag and applying the correct charset on decode in the various parser functions). I'm surprised that this hasn't been reported before.
How about we change this issue to "utf8 is not correctly decoded when parsing binary messages with utf=True"? We can still discuss whether or not my fix in #142517 was correct or not, since the correct context for that discussion is with this parsing issue resolved.
bitdancer commentedon Jan 12, 2026
Oh, and also, the other reason for this bug surfacing is the incorrect decision I made originally to make the default value of 'refold_source' be 'long' instead of 'none'. I'd love to change that default, but it would have backward compatibility implications, so I'm not sure we can.
picnixz commentedon Jan 12, 2026
Release is tomorrow I think. And the backports were already merged right?
cc @hugovk
picnixz commentedon Jan 12, 2026
Oh I thought it was also a releae for 3.13 and 3.14 tomorrow. Ok, not a RB sorry.
hugovk commentedon Jan 12, 2026
3.15.0a4 tomorrow, 3.13.12 and 3.14.3 on 2026-02-03.
medmunds commentedon May 11, 2026
#99927 is a reasonably clean report of that (from 2022).
bitdancer commentedon May 11, 2026
OK, good. Given that my fix was merged and backported, we can close this issue then.