Skip to content

Commit 96fd54e

Browse files
bitdancerbitdancer
committed
#4661: add bytes parsing and generation to email (email version bump to 5.1.0)
The work on this is not 100% complete, but everything is present to allow real-world testing of the code. The only remaining major todo item is to (hopefully!) enhance the handling of non-ASCII bytes in headers converted to unicode by RFC2047 encoding them rather than replacing them with '?'s.
1 parent 59fdd67 commit 96fd54e

11 files changed

Lines changed: 708 additions & 85 deletions

File tree

‎Doc/library/email.generator.rst‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ the Generator on a :class:`~email.message.Message` constructed by program may
2222
result in changes to the :class:`~email.message.Message` object as defaults are
2323
filled in.
2424

25+
:class:`bytes` output can be generated using the :class:`BytesGenerator` class.
26+
If the message object structure contains non-ASCII bytes, this generator's
27+
:meth:`~BytesGenerator.flatten` method will emit the original bytes. Parsing a
28+
binary message and then flattening it with :class:`BytesGenerator` should be
29+
idempotent for standards compliant messages.
30+
2531
Here are the public methods of the :class:`Generator` class, imported from the
2632
:mod:`email.generator` module:
2733

@@ -65,6 +71,13 @@ Here are the public methods of the :class:`Generator` class, imported from the
6571

6672
Note that for subparts, no envelope header is ever printed.
6773

74+
Messages parsed with a Bytes parser that have a
75+
:mailheader:`Content-Transfer-Encoding` of 8bit will be converted to a
76+
use a 7bit Content-Transfer-Encoding. Any other non-ASCII bytes in the
77+
message structure will be converted to '?' characters.
78+
79+
.. versionchanged:: 3.2 added support for re-encoding 8bit message bodies.
80+
6881
.. method:: clone(fp)
6982

7083
Return an independent clone of this :class:`Generator` instance with the
@@ -76,11 +89,27 @@ Here are the public methods of the :class:`Generator` class, imported from the
7689
:class:`Generator`'s constructor. This provides just enough file-like API
7790
for :class:`Generator` instances to be used in the :func:`print` function.
7891

79-
As a convenience, see the methods :meth:`Message.as_string` and
80-
``str(aMessage)``, a.k.a. :meth:`Message.__str__`, which simplify the generation
81-
of a formatted string representation of a message object. For more detail, see
92+
As a convenience, see the :class:`~email.message.Message` methods
93+
:meth:`~email.message.Message.as_string` and ``str(aMessage)``, a.k.a.
94+
:meth:`~email.message.Message.__str__`, which simplify the generation of a
95+
formatted string representation of a message object. For more detail, see
8296
:mod:`email.message`.
8397

98+
.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, fmt=None)
99+
100+
This class has the same API as the :class:`Generator` class, except that
101+
*outfp* must be a file like object that will accept :class`bytes` input to
102+
its `write` method. If the message object structure contains non-ASCII
103+
bytes, this generator's :meth:`~BytesGenerator.flatten` method will produce
104+
them as-is, including preserving parts with a
105+
:mailheader:`Content-Transfer-Encoding` of ``8bit``.
106+
107+
Note that even the :meth:`write` method API is identical: it expects
108+
strings as input, and converts them to bytes by encoding them using
109+
the ASCII codec.
110+
111+
.. versionadded:: 3.2
112+
84113
The :mod:`email.generator` module also provides a derived class, called
85114
:class:`DecodedGenerator` which is like the :class:`Generator` base class,
86115
except that non-\ :mimetype:`text` parts are substituted with a format string

‎Doc/library/email.message.rst‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,17 @@ Here are the methods of the :class:`Message` class:
111111
be decoded if this header's value is ``quoted-printable`` or ``base64``.
112112
If some other encoding is used, or :mailheader:`Content-Transfer-Encoding`
113113
header is missing, or if the payload has bogus base64 data, the payload is
114-
returned as-is (undecoded). If the message is a multipart and the
115-
*decode* flag is ``True``, then ``None`` is returned. The default for
116-
*decode* is ``False``.
114+
returned as-is (undecoded). In all cases the returned value is binary
115+
data. If the message is a multipart and the *decode* flag is ``True``,
116+
then ``None`` is returned.
117+
118+
When *decode* is ``False`` (the default) the body is returned as a string
119+
without decoding the :mailheader:`Content-Transfer-Encoding`. However,
120+
for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made
121+
to decode the original bytes using the `charset` specified by the
122+
:mailheader:`Content-Type` header, using the `replace` error handler. If
123+
no `charset` is specified, or if the `charset` given is not recognized by
124+
the email package, the body is decoded using the default ASCII charset.
117125

118126

119127
.. method:: set_payload(payload, charset=None)
@@ -160,6 +168,10 @@ Here are the methods of the :class:`Message` class:
160168
Note that in all cases, any envelope header present in the message is not
161169
included in the mapping interface.
162170

171+
In a model generated from bytes, any header values that (in contravention
172+
of the RFCs) contain non-ASCII bytes will have those bytes transformed
173+
into '?' characters when the values are retrieved through this interface.
174+
163175

164176
.. method:: __len__()
165177

‎Doc/library/email.parser.rst‎

Lines changed: 65 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,14 @@ Here is the API for the :class:`FeedParser`:
8080
if you feed more data to a closed :class:`FeedParser`.
8181

8282

83+
.. class:: BytesFeedParser(_factory=email.message.Message)
84+
85+
Works exactly like :class:`FeedParser` except that the input to the
86+
:meth:`~FeedParser.feed` method must be bytes and not string.
87+
88+
.. versionadded:: 3.2
89+
90+
8391
Parser class API
8492
^^^^^^^^^^^^^^^^
8593

@@ -131,33 +139,86 @@ class.
131139

132140
Similar to the :meth:`parse` method, except it takes a string object
133141
instead of a file-like object. Calling this method on a string is exactly
134-
equivalent to wrapping *text* in a :class:`StringIO` instance first and
142+
equivalent to wrapping *text* in a :class:`~io.StringIO` instance first and
135143
calling :meth:`parse`.
136144

137145
Optional *headersonly* is a flag specifying whether to stop parsing after
138146
reading the headers or not. The default is ``False``, meaning it parses
139147
the entire contents of the file.
140148

141149

150+
.. class:: BytesParser(_class=email.message.Message, strict=None)
151+
152+
This class is exactly parallel to :class:`Parser`, but handles bytes input.
153+
The *_class* and *strict* arguments are interpreted in the same way as for
154+
the :class:`Parser` constructor. *strict* is supported only to make porting
155+
code easier; it is deprecated.
156+
157+
.. method:: parse(fp, headeronly=False)
158+
159+
Read all the data from the binary file-like object *fp*, parse the
160+
resulting bytes, and return the message object. *fp* must support
161+
both the :meth:`readline` and the :meth:`read` methods on file-like
162+
objects.
163+
164+
The bytes contained in *fp* must be formatted as a block of :rfc:`2822`
165+
style headers and header continuation lines, optionally preceded by a
166+
envelope header. The header block is terminated either by the end of the
167+
data or by a blank line. Following the header block is the body of the
168+
message (which may contain MIME-encoded subparts, including subparts
169+
with a :mailheader:`Content-Transfer-Encoding` of ``8bit``.
170+
171+
Optional *headersonly* is a flag specifying whether to stop parsing after
172+
reading the headers or not. The default is ``False``, meaning it parses
173+
the entire contents of the file.
174+
175+
.. method:: parsebytes(bytes, headersonly=False)
176+
177+
Similar to the :meth:`parse` method, except it takes a byte string object
178+
instead of a file-like object. Calling this method on a byte string is
179+
exactly equivalent to wrapping *text* in a :class:`~io.BytesIO` instance
180+
first and calling :meth:`parse`.
181+
182+
Optional *headersonly* is as with the :meth:`parse` method.
183+
184+
.. versionadded:: 3.2
185+
186+
142187
Since creating a message object structure from a string or a file object is such
143-
a common task, two functions are provided as a convenience. They are available
188+
a common task, four functions are provided as a convenience. They are available
144189
in the top-level :mod:`email` package namespace.
145190

146191
.. currentmodule:: email
147192

148-
.. function:: message_from_string(s[, _class][, strict])
193+
.. function:: message_from_string(s, _class=email.message.Message, strict=None)
149194

150195
Return a message object structure from a string. This is exactly equivalent to
151196
``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted as
152197
with the :class:`Parser` class constructor.
153198

199+
.. function:: message_from_bytes(s, _class=email.message.Message, strict=None)
200+
201+
Return a message object structure from a byte string. This is exactly
202+
equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and
203+
*strict* are interpreted as with the :class:`Parser` class constructor.
204+
205+
.. versionadded:: 3.2
154206

155-
.. function:: message_from_file(fp[, _class][, strict])
207+
.. function:: message_from_file(fp, _class=email.message.Message, strict=None)
156208

157209
Return a message object structure tree from an open :term:`file object`.
158210
This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class*
159211
and *strict* are interpreted as with the :class:`Parser` class constructor.
160212

213+
.. function:: message_from_binary_file(fp, _class=email.message.Message, strict=None)
214+
215+
Return a message object structure tree from an open binary :term:`file
216+
object`. This is exactly equivalent to ``BytesParser().parse(fp)``.
217+
Optional *_class* and *strict* are interpreted as with the :class:`Parser`
218+
class constructor.
219+
220+
.. versionadded:: 3.2
221+
161222
Here's an example of how you might use this at an interactive Python prompt::
162223

163224
>>> import email

‎Doc/library/email.rst‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
email messages, including MIME documents.
77
.. moduleauthor:: Barry A. Warsaw <barry@python.org>
88
.. sectionauthor:: Barry A. Warsaw <barry@python.org>
9-
.. Copyright (C) 2001-2007 Python Software Foundation
9+
.. Copyright (C) 2001-2010 Python Software Foundation
1010
1111
1212
The :mod:`email` package is a library for managing email messages, including
@@ -92,6 +92,44 @@ table also describes the Python compatibility of each version of the package.
9292
+---------------+------------------------------+-----------------------+
9393
| :const:`4.0` | Python 2.5 | Python 2.3 to 2.5 |
9494
+---------------+------------------------------+-----------------------+
95+
| :const:`5.0` | Python 3.0 and Python 3.1 | Python 3.0 to 3.2 |
96+
+---------------+------------------------------+-----------------------+
97+
| :const:`5.1` | Python 3.2 | Python 3.0 to 3.2 |
98+
+---------------+------------------------------+-----------------------+
99+
100+
Here are the major differences between :mod:`email` version 5.1 and
101+
version 5.0:
102+
103+
* It is once again possible to parse messages containing non-ASCII bytes,
104+
and to reproduce such messages if the data containing the non-ASCII
105+
bytes is not modified.
106+
107+
* New functions :func:`message_from_bytes` and :func:`message_from_binary_file`,
108+
and new classes :class:`~email.parser.BytesFeedParser` and
109+
:class:`~email.parser.BytesParser` allow binary message data to be parsed
110+
into model objects.
111+
112+
* Given bytes input to the model, :meth:`~email.message.Message.get_payload`
113+
will by default decode a message body that has a
114+
:mailheader:`Content-Transfer-Encoding` of `8bit` using the charset specified
115+
in the MIME headers and return the resulting string.
116+
117+
* Given bytes input to the model, :class:`~email.generator.Generator` will
118+
convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
119+
8bit to instead have a 7bit Content-Transfer-Encoding.
120+
121+
* New function :class:`~email.generator.BytesGenerator` produces bytes
122+
as output, preserving any unchanged non-ASCII data that was
123+
present in the input used to build the model, including message bodies
124+
with a :mailheader:`Content-Transfer-Encoding` of 8bit.
125+
126+
Here are the major differences between :mod:`email` version 5.0 and version 4:
127+
128+
* All operations are on unicode strings. Text inputs must be strings,
129+
text outputs are strings. Outputs are limited to the ASCII character
130+
set and so can be encoded to ASCII for transmission. Inputs are also
131+
limited to ASCII; this is an acknowledged limitation of email 5.0 and
132+
means it can only be used to parse email that is 7bit clean.
95133

96134
Here are the major differences between :mod:`email` version 4 and version 3:
97135

‎Lib/email/__init__.py‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
"""A package for parsing, handling, and generating email messages."""
66

7-
__version__ = '5.0.0'
7+
__version__ = '5.1.0'
88

99
__all__ = [
1010
'base64mime',
@@ -16,7 +16,9 @@
1616
'iterators',
1717
'message',
1818
'message_from_file',
19+
'message_from_binary_file',
1920
'message_from_string',
21+
'message_from_bytes',
2022
'mime',
2123
'parser',
2224
'quoprimime',
@@ -36,6 +38,13 @@ def message_from_string(s, *args, **kws):
3638
from email.parser import Parser
3739
return Parser(*args, **kws).parsestr(s)
3840

41+
def message_from_bytes(s, *args, **kws):
42+
"""Parse a bytes string into a Message object model.
43+
44+
Optional _class and strict are passed to the Parser constructor.
45+
"""
46+
from email.parser import BytesParser
47+
return BytesParser(*args, **kws).parsebytes(s)
3948

4049
def message_from_file(fp, *args, **kws):
4150
"""Read a file and parse its contents into a Message object model.
@@ -44,3 +53,11 @@ def message_from_file(fp, *args, **kws):
4453
"""
4554
from email.parser import Parser
4655
return Parser(*args, **kws).parse(fp)
56+
57+
def message_from_binary_file(fp, *args, **kws):
58+
"""Read a binary file and parse its contents into a Message object model.
59+
60+
Optional _class and strict are passed to the Parser constructor.
61+
"""
62+
from email.parser import Parser
63+
return BytesParser(*args, **kws).parse(fp)

‎Lib/email/feedparser.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,3 +482,10 @@ def _parse_headers(self, lines):
482482
if lastheader:
483483
# XXX reconsider the joining of folded lines
484484
self._cur[lastheader] = EMPTYSTRING.join(lastvalue).rstrip('\r\n')
485+
486+
487+
class BytesFeedParser(FeedParser):
488+
"""Like FeedParser, but feed accepts bytes."""
489+
490+
def feed(self, data):
491+
super().feed(data.decode('ascii', 'surrogateescape'))

0 commit comments

Comments
 (0)