http://hg.code.sf.net:8000/p/roundup/code/atom-log/tip/roundup/mailgw.py Mercurial Repository: p/roundup/code: roundup/mailgw.py history 2026-04-08T21:39:40-04:00 chore: remove __future print_funcion from code. http://hg.code.sf.net:8000/p/roundup/code/#changeset-9c3ec0a5c7fc88acb8a65632ecc13b2d52380314 John Rouillard rouilj@ieee.org 2026-04-08T21:39:40-04:00 2026-04-08T21:39:40-04:00
changeset 9c3ec0a5c7fc
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description chore: remove __future print_funcion from code.

Not needed as of Python 3.
files
feat: improve store_trace_reason with extract parameter http://hg.code.sf.net:8000/p/roundup/code/#changeset-f80c566f5726c55c8654ec24a9f73a138b2c8a6f John Rouillard rouilj@ieee.org 2026-04-06T22:10:23-04:00 2026-04-06T22:10:23-04:00
changeset f80c566f5726
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: improve store_trace_reason with extract parameter

store_trace_reason() used embedded code to extract reason based on the
location passed to the function.

This change adds support for extract keyword that is a Python
expression eval'ed when the underlying function/method is called. All
callers now set the extract parameter. The prior embedded code has
been removed from store_trace_reason().

Failure to eval the expression results in an roundup.logcontext error
severity log. Also updated docs.

Also replaced env['REQUEST_URI'] with env['PATH_INFO'] for web based
entry points as REQUEST_URI isn't documented as a required key and
some other front end (e.g. zope, cgi) might not supply this.
files
feature: add thread local trace_id and trace_reason to logging. http://hg.code.sf.net:8000/p/roundup/code/#changeset-14c7c07b32d846e405260147a3112e8c8b15642a John Rouillard rouilj@ieee.org 2025-09-16T22:53:00-04:00 2025-09-16T22:53:00-04:00
changeset 14c7c07b32d8
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feature: add thread local trace_id and trace_reason to logging.

Added trace_id to default logging so that all logs for a given request
share the same trace_id.

This allows correlation of logs across a request.

admin_guide.txt, upgrading.txt:
add docs
update sample configs to include trace_id.
rewrite logging docs in admin_guide. Hopefully they are clearer now.
clean up some stuff in the logging config file docs.

admin.py:

add decorators to run_command to enable trace_id.
change calls to db.commit() to use run_command to get trace_id.

configuration.py:

clean up imports.
update docstrings, comments and inline docs.
add trace_id to default log format.
add function for testing decorated with trace_id.
add support for dumping stack trace in logging.
add check for pytest in sys.modules to enable log propagation when
pytest is running. Otherwise tests fail as the caplog logger doesn't
see the roundup logs.

logcontext.py:

new file to handle thread local contextvar mangement.

mailgw.py:

add decorators for trace_id etc.

scripts/roundup_xlmrpc_server.py:

add decorators for trace_id etc.
fix encoding bug turning bytes into a string.
fix command line issue where we can't set encoding. (not sure if
changing encoding via command line even works)

cgi/client.py

decorate two entry points for trace_id etc.

cgi/wsgi_handler.py:

decorate entry point for trace_id etc.

test/test_config.py:

add test for trace_id in new log format.
test various cases for sinfo and errors in formating msg.
files
flake8 fixes http://hg.code.sf.net:8000/p/roundup/code/#changeset-57f34b0b912cf5dea9cd93e5536878ca0ce39fb3 John Rouillard rouilj@ieee.org 2023-05-07T09:10:10-04:00 2023-05-07T09:10:10-04:00
changeset 57f34b0b912c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description flake8 fixes

move import to top, indentation, remove trailing whitespace ...
files
issue2551262 - make mail gateway subject prefix parsing accept spaces http://hg.code.sf.net:8000/p/roundup/code/#changeset-86862ed039fa659bb2e604f8a5d8329b8ecb3e77 John Rouillard rouilj@ieee.org 2023-03-21T19:46:38-04:00 2023-03-21T19:46:38-04:00
changeset 86862ed039fa
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551262 - make mail gateway subject prefix parsing accept spaces

Allow spaces before/after prefix. Also allow spaces between classname
and id number in prefix designator. So "[ issue 23 ] subject" is
parsed like "[issue23] subject".
files
flake8 spacing changes, split multiple imports http://hg.code.sf.net:8000/p/roundup/code/#changeset-8e118eb20d86a96a84d294febf8eb32d48453273 John Rouillard rouilj@ieee.org 2023-03-19T12:26:42-04:00 2023-03-19T12:26:42-04:00
changeset 8e118eb20d86
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description flake8 spacing changes, split multiple imports
files
chore: flake8 whitespace fixes http://hg.code.sf.net:8000/p/roundup/code/#changeset-357e185752ed327e928835b78973654dd9a4bc0c John Rouillard rouilj@ieee.org 2023-02-26T10:40:25-05:00 2023-02-26T10:40:25-05:00
changeset 357e185752ed
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description chore: flake8 whitespace fixes
files
Put oauth client secret into a file http://hg.code.sf.net:8000/p/roundup/code/#changeset-f918351a0fe6c5d830bc3c736b385791f2535832 Ralf Schlatterbeck rsc@runtux.com 2022-11-26T11:32:12+01:00 2022-11-26T11:32:12+01:00
changeset f918351a0fe6
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Put oauth client secret into a file

.. and get rid of the -I option of the mailgw script.
files
Add OAuth authentication to the mailgw script http://hg.code.sf.net:8000/p/roundup/code/#changeset-3359dc1dabb09c4594c730feec4eafceb323186f Ralf Schlatterbeck rsc@runtux.com 2022-11-23T10:13:48+01:00 2022-11-23T10:13:48+01:00
changeset 3359dc1dabb0
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add OAuth authentication to the mailgw script

Now IMAPS can be used with OAuth as required by several large cloud
providers. Move command line processing of the mailgw script to
``argparse``. Note that the command line options of the mailgw have
changed, see upgrading.txt for details.
files
issue2551232 - modify in-reply-to threading when multiple matches http://hg.code.sf.net:8000/p/roundup/code/#changeset-bd2c3b2010c33483301a75b52047d52277a5e289 John Rouillard rouilj@ieee.org 2022-09-08T18:49:46-04:00 2022-09-08T18:49:46-04:00
changeset bd2c3b2010c3
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551232 - modify in-reply-to threading when multiple matches

if an email is missing an issue designator, in-reply-to threading is
attempted. In this change if in-reply-to threading matches multiple
issues, fall back to matching on subject. It used to just arbitrairly
choose the first matching issue.
files
flake8 changes. http://hg.code.sf.net:8000/p/roundup/code/#changeset-3d2ec36541b9c643a89c4f35755d9d222b6f8dfd John Rouillard rouilj@ieee.org 2022-09-08T15:28:36-04:00 2022-09-08T15:28:36-04:00
changeset 3d2ec36541b9
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description flake8 changes.

Also identified a possible crash when a message arrives without an
issue designator. https://issues.roundup-tracker.org/issue2551232.
files
Add more translation tests for mailgw.py http://hg.code.sf.net:8000/p/roundup/code/#changeset-567283742a5c3b57cce6bc7f27f249dfec74036f John Rouillard rouilj@ieee.org 2022-05-12T14:50:50-04:00 2022-05-12T14:50:50-04:00
changeset 567283742a5c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Add more translation tests for mailgw.py

Verify error in German for [issue999999] and [frobulated] in subject
line. The issue doesn't exist and the class doesn't exist.

Also add warning to mailgw.py for mailgw.py::handle_message that any
chnags to that method need to be replicated for all Translate tests in
test_mailgw.py since it can't be tested directly as it destroys the
mock used for the database by the test harness.
files
Testing translations in mailgw.py, update translations, i18n improved http://hg.code.sf.net:8000/p/roundup/code/#changeset-01216187a16781e67dcd674e0230d79f411442c0 John Rouillard rouilj@ieee.org 2022-05-11T21:20:34-04:00 2022-05-11T21:20:34-04:00
changeset 01216187a167
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Testing translations in mailgw.py, update translations, i18n improved

i18n.py: translation objects get a _file which is the file that catalog
was loaded from. Useful for debugging.

mailgw.py: fix code to include roundup.i18n and invoke propert method.

GNUMakefile: added roundup.pot target. Also building roundup.pot sets
the release version and product code in roundup.pot.
Release version from roundup.__init__.py.

merged roundup.pot changes into all *.po files. Not sure if the
version info in the headers of the .po files is supposed to be
updated to match roundup.pot or not.

test_mailgw.py: test cases for en and de translations tested. Not sure
why I neede to set roundupdb._ and mailgw._, but followed setUp
method code and it worked.
files
Add i18n object to roundupdb.Database http://hg.code.sf.net:8000/p/roundup/code/#changeset-408fd477761fa40fd7dfa13c4a25be572cd46dd9 Ralf Schlatterbeck rsc@runtux.com 2022-05-03T14:51:59+02:00 2022-05-03T14:51:59+02:00
changeset 408fd477761f
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add i18n object to roundupdb.Database

This makes the i18n object accessible everywhere (including in detectors
where localized error messages were impossible). See issue2551184
files
issue2551186 - replace socket.sslerror in mailgw.py. http://hg.code.sf.net:8000/p/roundup/code/#changeset-3e8f2104753b6431ddc593a5c89506687564214b John Rouillard rouilj@ieee.org 2022-01-04T18:39:40-05:00 2022-01-04T18:39:40-05:00
changeset 3e8f2104753b
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551186 - replace socket.sslerror in mailgw.py.

roundup/mailgw.py:

replaced socket.sslerror with anypy/ssl_.SSLError
also looks like a socket.sslerror was removed from pops handling.
added it back using method above.

roundupanypy/ssl_.py:
defines replacement SSLError suitable for python2 or 3

tested by running nc -lp 995 or 993 (pop3s/imaps) and sending
gibberish when mailgw connects. This generates a bad version number
SSLError.

I need to get my imap and pop mock servers included for testing at
some point, but I am not sure how to make them bind to the right port
as they are priv ports.
files
Fix filename created from mail attachments http://hg.code.sf.net:8000/p/roundup/code/#changeset-14c9284a8bad18fb1cfa50aa7e5c9d54a0522b4b Ralf Schlatterbeck rsc@runtux.com 2021-03-17T10:54:14+01:00 2021-03-17T10:54:14+01:00
changeset 14c9284a8bad
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix filename created from mail attachments

Fixes issue2551118
files
Handle auto-submitted header on inbound email like bulk. http://hg.code.sf.net:8000/p/roundup/code/#changeset-ca0915457761df303d6da761287bcf0780dd9b68 John Rouillard rouilj@ieee.org 2020-07-28T20:58:04-04:00 2020-07-28T20:58:04-04:00
changeset ca0915457761
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Handle auto-submitted header on inbound email like bulk.

Part of issue2550564.
files
Fix broken import http://hg.code.sf.net:8000/p/roundup/code/#changeset-80915fd9ad249633e64664d164b14126c3e2e786 John Rouillard rouilj@ieee.org 2020-04-03T18:31:35-04:00 2020-04-03T18:31:35-04:00
changeset 80915fd9ad24
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix broken import

apparently when running run_test.py in my directory

from exceptions

somehow works but not when run in CI.
files
issue2551083 Replace BaseException and Exception with RoundupException http://hg.code.sf.net:8000/p/roundup/code/#changeset-c177e7128dc987e64d2a829d2dc81dc4538da17e John Rouillard rouilj@ieee.org 2020-04-03T16:18:59-04:00 2020-04-03T16:18:59-04:00
changeset c177e7128dc9
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551083 Replace BaseException and Exception with RoundupException

issue2551083 - Replace BaseException and Exception as base classes
with new RoundupException (inheriting from Exception) for most roundup
exceptions.
files
Flake8 fixes; remove unused imports; format changes. http://hg.code.sf.net:8000/p/roundup/code/#changeset-a328f9e3307bcb963298cada1ab2f59cac31bccb John Rouillard rouilj@ieee.org 2020-01-01T20:45:09-05:00 2020-01-01T20:45:09-05:00
changeset a328f9e3307b
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Flake8 fixes; remove unused imports; format changes.
files
Fix mailgw IMAP message handling http://hg.code.sf.net:8000/p/roundup/code/#changeset-7264b2e79a315107b9c8819671123fb2c3049494 Ralf Schlatterbeck rsc@runtux.com 2019-10-09T18:18:22+02:00 2019-10-09T18:18:22+02:00
changeset 7264b2e79a31
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix mailgw IMAP message handling
files
[issue2551008] Handle the issue in _decode_header by decoding only when decode_header returns bytes. http://hg.code.sf.net:8000/p/roundup/code/#changeset-611737bc72616122bd54519e92c31c0104852e36 Ezio Melotti ezio.melotti@gmail.com 2019-06-05T00:13:45+02:00 2019-06-05T00:13:45+02:00
changeset 611737bc7261
branch issue2551008
bookmark
tag
user Ezio Melotti <ezio.melotti@gmail.com>
description Handle the issue in _decode_header by decoding only when decode_header returns bytes.
files
[issue2551008] Remove trailing whitespace. http://hg.code.sf.net:8000/p/roundup/code/#changeset-945051a69d85a53177b1e39c8fe3be05d0711076 Ezio Melotti ezio.melotti@gmail.com 2019-06-04T23:41:45+02:00 2019-06-04T23:41:45+02:00
changeset 945051a69d85
branch issue2551008
bookmark
tag
user Ezio Melotti <ezio.melotti@gmail.com>
description Remove trailing whitespace.
files
[issue2551008] working branch for fixing https://issues.roundup-tracker.org/issue2551008 http://hg.code.sf.net:8000/p/roundup/code/#changeset-cacef71b3a5493201d6752341a00adbc4d351697 John Rouillard rouilj@ieee.org 2019-06-03T20:50:58-04:00 2019-06-03T20:50:58-04:00
changeset cacef71b3a54
branch issue2551008
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description working branch for fixing https://issues.roundup-tracker.org/issue2551008
files
Do not transcode binary email attachments (issue2551004). http://hg.code.sf.net:8000/p/roundup/code/#changeset-081be318661bd712500821e51eb2ae681732570a Joseph Myers jsm@polyomino.org.uk 2018-09-26T01:24:19+00:00 2018-09-26T01:24:19+00:00
changeset 081be318661b
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Do not transcode binary email attachments (issue2551004).

Only transcode attachments if a charset is specified or if they have a
text/* content-type.
files
Fix email interfaces with Python 3 (issue 2550974, issue 2551000). http://hg.code.sf.net:8000/p/roundup/code/#changeset-29346d92d80cdfb50a30186dc1e45414dcdecf56 Joseph Myers jsm@polyomino.org.uk 2018-09-16T13:55:53+00:00 2018-09-16T13:55:53+00:00
changeset 29346d92d80c
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Fix email interfaces with Python 3 (issue 2550974, issue 2551000).

This patch fixes various issues handling incoming email with
roundup-mailgw with Python 3.

Incoming email must always be handled as bytes, not strings, because
it may contain 8-bit-encoded MIME parts with different encodings in
each part. When handling piped input, that means using
sys.stdin.buffer in Python 3 for binary input, along with
message_from_binary_file, not sys.stdin which is text input and may be
for the wrong encoding and not message_from_file. (In turn, tests
that use MailGW.main with text input are affected so an s2b call is
inserted in the test code and it is made to use BytesIO not StringIO.
Properly all the test messages in test_mailgw.py ought to use b''
explicitly rather than having such an s2b conversion, and there ought
to be test messages using 8-bit encodings with non-ASCII characters to
verify that that case works.)

imaplib and poplib return bytes not strings with Python 3 (from
inspection of the code, not tested), as is necessary for the above
reasons. Thus, the handling of IMAP and POP messages must expect
bytes and handle the data accordingly.

For messages from mailboxes, I saw the same problem described in issue
2551000 for a multipart message with a single (non-ASCII) part. The
Roundup code requires RoundupMessage not email.message.Message to be
used recursively for all MIME parts of a message. Because the mailbox
module uses email.message_from_* directly without passing the _class
argument to them, fixing this requires temporarily patching the email
module to ensure _class=RoundupMessage gets passed to those methods.
files
use gpg module instead of pyme module for PGP encryption http://hg.code.sf.net:8000/p/roundup/code/#changeset-b7fa56ced601c5293f7a5d735eaf193973a2f682 Christof Meerwald cmeerw@cmeerw.org 2018-08-12T16:17:14+01:00 2018-08-12T16:17:14+01:00
changeset b7fa56ced601
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description use gpg module instead of pyme module for PGP encryption
files
updated mailgw to no longer use mimetools based on jerrykan's patch http://hg.code.sf.net:8000/p/roundup/code/#changeset-725266c03eab7b5df86fa532c40123826ee7fc2b Christof Meerwald cmeerw@cmeerw.org 2018-08-12T16:15:10+01:00 2018-08-12T16:15:10+01:00
changeset 725266c03eab
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description updated mailgw to no longer use mimetools based on jerrykan's patch
files
reworked random number use http://hg.code.sf.net:8000/p/roundup/code/#changeset-52cb53eedf777c4c915a89f44a5059285039b6e0 Christof Meerwald cmeerw@cmeerw.org 2018-08-04T22:40:16+01:00 2018-08-04T22:40:16+01:00
changeset 52cb53eedf77
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description reworked random number use
prefer secrets module from Python 3.6+, random.SystemRandom and finally plain random
files
only conditionally import mimetools http://hg.code.sf.net:8000/p/roundup/code/#changeset-1c9a7310e4f0a65c554a44789206868037452f36 Christof Meerwald cmeerw@cmeerw.org 2018-07-28T14:11:18+01:00 2018-07-28T14:11:18+01:00
changeset 1c9a7310e4f0
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description only conditionally import mimetools
files
Python 3 preparation: avoid string.split(). http://hg.code.sf.net:8000/p/roundup/code/#changeset-86b6cea7a9752bc3caecd69f924796c73e904cc0 Joseph Myers jsm@polyomino.org.uk 2018-07-25T10:42:42+00:00 2018-07-25T10:42:42+00:00
changeset 86b6cea7a975
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: avoid string.split().
files
Python 3 preparation: StringIO. http://hg.code.sf.net:8000/p/roundup/code/#changeset-55f09ca366c484e1c06dd34a4efead3f1634d7d7 Joseph Myers jsm@polyomino.org.uk 2018-07-25T09:08:29+00:00 2018-07-25T09:08:29+00:00
changeset 55f09ca366c4
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: StringIO.

This generally arranges for StringIO and cStringIO references to use
io.StringIO for Python 3 but io.BytesIO for Python 2, consistent with
the string representations generally used in Roundup. A special
FasterStringIO in the TAL code, which referenced internals of the old
Python 2 StringIO module, is cut down so it doesn't actually do
anything beyond the StringIO class it inherits from (it would also be
reasonable to remove FasterStringIO completely). One place in
roundup_server.py clearly needing binary I/O is made to use io.BytesIO
unconditionally.
files
Python 3 preparation: unicode. http://hg.code.sf.net:8000/p/roundup/code/#changeset-56c9bcdea47f22412e4f0768775d1abea52d19c2 Joseph Myers jsm@polyomino.org.uk 2018-07-25T09:05:58+00:00 2018-07-25T09:05:58+00:00
changeset 56c9bcdea47f
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: unicode.

This patch introduces roundup/anypy/strings.py, which has a comment
explaining the string representations generally used and common
functions to handle the required conversions. Places in the code that
explicitly reference the "unicode" type / built-in function are
generally changed to use the new functions (or, in a few places where
those new functions don't seem to fit well, other approaches such as
references to type(u'') or use of the codecs module). This patch does
not generally attempt to address text conversions in any places not
currently referencing the "unicode" type (although
scripts/import_sf.py is made to use binary I/O in places as fixing the
"unicode" reference didn't seem coherent otherwise).
files
Python 3 preparation: replace raw_input uses. http://hg.code.sf.net:8000/p/roundup/code/#changeset-4cf48ff01e048774ed1f762dcfc8935dbb11faa9 Joseph Myers jsm@polyomino.org.uk 2018-07-24T23:47:10+00:00 2018-07-24T23:47:10+00:00
changeset 4cf48ff01e04
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: replace raw_input uses.

The existing my_input in roundup/admin.py is moved to a new
roundup/anypy/my_input.py, which is then used in more places. Manual
patch.
files
Python 3 preparation: update next() usage for iterators. http://hg.code.sf.net:8000/p/roundup/code/#changeset-831787cf669468fdff1d0d4ff9b5a90cf6285515 Joseph Myers jsm@polyomino.org.uk 2018-07-24T23:12:32+00:00 2018-07-24T23:12:32+00:00
changeset 831787cf6694
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: update next() usage for iterators.

Tool-assisted patch. Note that various classes in TAL code with
next() methods are not actually Python iterators and so are not
changed in this patch, but roundup/cgi/ZTUtils/Iterator.py includes
the IterIter class which converts between the two styles of iterator.
files
Python 3 preparation: make relative imports explicit. http://hg.code.sf.net:8000/p/roundup/code/#changeset-d26921b851c328681aa6638475ce2cdae29963b4 Joseph Myers jsm@polyomino.org.uk 2018-07-24T22:22:08+00:00 2018-07-24T22:22:08+00:00
changeset d26921b851c3
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: make relative imports explicit.

Tool-generated patch.
files
Python 3 preparation: change "x.has_key(y)" to "y in x". http://hg.code.sf.net:8000/p/roundup/code/#changeset-0942fe89e82e3e84d95a3d0ca5d8d15502f57348 Joseph Myers jsm@polyomino.org.uk 2018-07-24T22:08:17+00:00 2018-07-24T22:08:17+00:00
changeset 0942fe89e82e
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: change "x.has_key(y)" to "y in x".

(Also likewise "not in" where appropriate.) Tool-generated patch.
files
Python 3 preparation: "raise" syntax. http://hg.code.sf.net:8000/p/roundup/code/#changeset-35ea9b1efc1494f29fb07690e7087686de58e0f3 Joseph Myers jsm@polyomino.org.uk 2018-07-24T21:39:58+00:00 2018-07-24T21:39:58+00:00
changeset 35ea9b1efc14
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: "raise" syntax.

Changing "raise Exception, value" to "raise Exception(value)".
Tool-assisted patch. Particular cases to check carefully are the one
place in frontends/ZRoundup/ZRoundup.py where a string exception
needed to be fixed, and the one in roundup/cgi/client.py involving
raising an exception with a traceback (requires three-argument form of
raise in Python 2, which as I understand it requires exec() to avoid a
Python 3 syntax error).
files
Python 3 preparation: convert print to a function. http://hg.code.sf.net:8000/p/roundup/code/#changeset-64b05e24dbd889f52bf8f773d3456bd0135baa27 Joseph Myers jsm@polyomino.org.uk 2018-07-24T09:54:52+00:00 2018-07-24T09:54:52+00:00
changeset 64b05e24dbd8
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: convert print to a function.

Tool-assisted patch. It is possible that some "from __future__ import
print_function" are not in fact needed, if a file only uses print()
with a single string as an argument and so would work fine in Python 2
without that import.
files
A real fix for the problem where: http://hg.code.sf.net:8000/p/roundup/code/#changeset-91954be46a66ab9f301cfbabf6cfb9a269d324dd John Rouillard rouilj@ieee.org 2018-07-08T11:34:42-04:00 2018-07-08T11:34:42-04:00
changeset 91954be46a66
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description A real fix for the problem where:

import random

would result in every call to random() returning the same value
in the web interface.

While cgi/client.py:Client::__init.py__ was calling random.seed(),
on most systems random was SystemRandom and not the default random.

As a result the random as you would get from:

import random

was never being seeded. I added a function to access and seed the
random bound instance of random.Random that is called during init.

This fixes all three places where I saw the broken randomness.
It should also fix:

http://psf.upfronthosting.co.za/roundup/meta/issue644

I also removed the prior code that would bail if systemRandom was not
available.
files
Force all uses of random to use SystemRandom and abort if http://hg.code.sf.net:8000/p/roundup/code/#changeset-66a17c80e03503a503efa70475e918c1265ced80 John Rouillard rouilj@ieee.org 2018-07-07T22:02:41-04:00 2018-07-07T22:02:41-04:00
changeset 66a17c80e035
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Force all uses of random to use SystemRandom and abort if
pseudorandom random.Random would be used rather than
Random.SystemRandom.

random.Random is returning the same value time after time. Even when
being seeded after instantiation, calls to the random.random()
function return the same value like it's not advanceing the state of
the generator.

So "fix" is to force use of system random generator to generate:

one time keys for password reset (action.py)
random passwords when resetting passwords (password.py)
serial number for auto ssl cert generation (roundup_server.py)
Message-ID's in email: mailgw.py, client.py
anti-csrf nonces (templating.py)
files
Work around a line-length limit in poplib http://hg.code.sf.net:8000/p/roundup/code/#changeset-57caeefb2f811764a27b8b41ae70c42d1943479c Ralf Schlatterbeck rsc@runtux.com 2018-06-07T12:39:31+02:00 2018-06-07T12:39:31+02:00
changeset 57caeefb2f81
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Work around a line-length limit in poplib

Work around a limitation in python2.7 implementation of poplib (for the
pop3 protocol for fetching emails): It seems poplib applies a
line-length limit not just to the lines involving the pop3 protocol but
to any email content, too. This sometimes leads to tracebacks whenever
an email exceeding this limit is encountered. We "fix" this by
monkey-patching poplib with a larger line-limit. Thanks to Heiko
Stegmann for discovering this.
files
Codecov showed text/html followed by text/plain not tested. Fixed bug http://hg.code.sf.net:8000/p/roundup/code/#changeset-91354bf0b6831f03fd47679eb975b7e58458755e John Rouillard rouilj@ieee.org 2017-10-13T22:56:42-04:00 2017-10-13T22:56:42-04:00
changeset 91354bf0b683
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Codecov showed text/html followed by text/plain not tested. Fixed bug
discovered.

The html part was double attached. Removed an unneeded attachment
call. Added some more test cases. Reworked test code to make selecting
html filter tool easier.
files
issue2550799: provide basic support for handling html only emails http://hg.code.sf.net:8000/p/roundup/code/#changeset-e20f472fde7d9ed7e1e5120de2941181efdaeba7 John Rouillard rouilj@ieee.org 2017-10-13T21:46:59-04:00 2017-10-13T21:46:59-04:00
changeset e20f472fde7d
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550799: provide basic support for handling html only emails

Initial implementation and testing with the dehtml html converter
done.

The use of beautifulsoup 4 is not tested. My test system breaks when
running dehtml.py using beautiful soup. I don't get the failures when
running under the test harness, but the text output is significantly
different (different line breaks, number of newlines etc.)

The tests for dehtml need to be generated for beautiful soup and the
expected output changed. Since I have a wonky install of beautiful
soup, I don't trust my output as the standard to test against. Also
since beautiful soup is optional, the test harness needs to skip the
beautifulsoup tests if import bs4 fails. Again something outside of my
expertise. I deleted the work I had done to implement that. I could
not get it working and wanted to get this feature in in some form.
files
Python 2 and 3 support. Convert Exception into BaseException in core code. http://hg.code.sf.net:8000/p/roundup/code/#changeset-32f95ec6bd8e058056dd9ab57d641a19f904281f John Rouillard rouilj@ieee.org 2017-09-04T21:56:49-04:00 2017-09-04T21:56:49-04:00
changeset 32f95ec6bd8e
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Python 2 and 3 support. Convert Exception into BaseException in core code.
files
Use Python-3-compatible 'as' syntax for except statements http://hg.code.sf.net:8000/p/roundup/code/#changeset-198b6e810c67c4e7e38f1aa41edbc864783fe6e6 Eric S. Raymond esr@thyrsus.com 2017-08-24T22:21:37-04:00 2017-08-24T22:21:37-04:00
changeset 198b6e810c67
branch
bookmark
tag
user Eric S. Raymond <esr@thyrsus.com>
description Use Python-3-compatible 'as' syntax for except statements

Many raise statements near these are also fixed.
So are two ivorrect file encoding marks ('utf8'->'utf-8').
files
Mailgw: Improves diagnostic messages for DEBUG. http://hg.code.sf.net:8000/p/roundup/code/#changeset-86190260f077fcbf59833116c99db9979741c8a4 Bernhard Reiter bernhard@intevation.de 2017-05-10T10:36:39+02:00 2017-05-10T10:36:39+02:00
changeset 86190260f077
branch
bookmark
tag
user Bernhard Reiter <bernhard@intevation.de>
description Mailgw: Improves diagnostic messages for DEBUG.

* Adds debug log messages for bounce mails to aid tracing of emails.
files
Fix subject parsing in mail gateway. http://hg.code.sf.net:8000/p/roundup/code/#changeset-6bbb6dd974589a52c5f1063c421b6456260fc0ba Ralf Schlatterbeck rsc@runtux.com 2016-07-19T11:47:53+02:00 2016-07-19T11:47:53+02:00
changeset 6bbb6dd97458
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix subject parsing in mail gateway.

The previous parsing routine would not ensure that arguments are at the
end of the subject and when subject_suffix_parsing was configured to be
'loose' it would truncate the subject when encountering a double prefix,
e.g. Subject: [frobulated] [frobulatedagain] this part would be lost
files
Fix issue934009: Have New Issues Submitted By Email *Not* Change Body! http://hg.code.sf.net:8000/p/roundup/code/#changeset-14abd0a6720774675b58f9e4e246c17dc96e49e7 John Rouillard rouilj@ieee.org 2016-07-02T20:05:06-04:00 2016-07-02T20:05:06-04:00
changeset 14abd0a67207
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix issue934009: Have New Issues Submitted By Email *Not* Change Body!

The mailgw config options: keep_quoted_text and leave_body_unchanged
can now have a new values: new. If set to new, keep_quoted_text acts
like yes if the message is starting a new issue. Otherise it strips
quoted text.

This allows somebody to start a new issue by forwarding
a threaded email (with multiple quoted parts) into roundup and
keeping all the quoted parts. If leave_body_unchanged is set to
new, even the signature on the email that starts a new issue will be
preserved.
files
issue2550823 Improve mailgw logging for node creation errors. Add a http://hg.code.sf.net:8000/p/roundup/code/#changeset-d3d9d1402763900a11049cea569e5e3a5f34c601 John Rouillard rouilj@ieee.org 2016-06-18T00:39:01-04:00 2016-06-18T00:39:01-04:00
changeset d3d9d1402763
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550823 Improve mailgw logging for node creation errors. Add a
call to logger.exception when an error is raised on creation
files
issue2550757: one bug raised by issue fixed. Patch created by http://hg.code.sf.net:8000/p/roundup/code/#changeset-d995ee7d49bf28ebd2d06944a68bbea0222e180a John Rouillard rouilj@ieee.org 2016-04-30T21:33:41-04:00 2016-04-30T21:33:41-04:00
changeset d995ee7d49bf
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550757: one bug raised by issue fixed. Patch created by
W. Trevor King (wking) for documentation of mailgw applied by
John Rouillard.
files
Fix issue1615201: Added a new configuration option EMAIL_KEEP_REAL_FROM http://hg.code.sf.net:8000/p/roundup/code/#changeset-a46d5d0fd5f83d3ef46be378f76cacce3feca681 Peter Funk pf@artcom-gmbh.de 2016-03-18T16:18:20+01:00 2016-03-18T16:18:20+01:00
changeset a46d5d0fd5f8
branch
bookmark
tag
user Peter Funk <pf@artcom-gmbh.de>
description Fix issue1615201: Added a new configuration option EMAIL_KEEP_REAL_FROM
which makes the mail gateway ignore a Resent-From:-header and use the
real From:-header of the original author for further processing of the
message. Setting this option to 'yes' restores the original behaviour
of Roundup before the change in version 0.7.0 where the processing of
Resent-From: was added unconditionally.
files
Remove rfc2822.py http://hg.code.sf.net:8000/p/roundup/code/#changeset-7f8f27ce1a060770634df717f1af6e843e0663d5 John Kristensen john@jerrykan.com 2014-05-11T01:05:58+10:00 2014-05-11T01:05:58+10:00
changeset 7f8f27ce1a06
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Remove rfc2822.py

A majority of the module is not being used, and the one function that is
being used can easily be replicated in mailgw.py, meaning that
everything related to the rfc2822 module can be removed.
files
Replace rfc822 imports with email package (issue2550870) http://hg.code.sf.net:8000/p/roundup/code/#changeset-13f8f88ad9846793162e4dff87c2b29ababe0efc John Kristensen john@jerrykan.com 2015-02-27T00:48:25+11:00 2015-02-27T00:48:25+11:00
changeset 13f8f88ad984
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Replace rfc822 imports with email package (issue2550870)

The rfc822 package has been deprecated since Python v2.3 in favour of
the email package.
files
Implementation for: http://hg.code.sf.net:8000/p/roundup/code/#changeset-6e9b9743de89d2acc152b890dc6848f811ffadce John Rouillard rouilj@ieee.org 2013-04-23T23:06:09-04:00 2013-04-23T23:06:09-04:00
changeset 6e9b9743de89
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Implementation for:

http://issues.roundup-tracker.org/issue2550731

Add mechanism for the detectors to be able to tell the source of the
data changes.

Support for tx_Source property on database handle. Can be
used by detectors to find out the source of a change in an auditor to
block changes arriving by unauthenticated mechanisms (e.g. plain email
where headers can be faked). The property db.tx_Source has the
following values:
* None - Default value set to None. May be valid if it's a script
that is created by the user. Otherwise it's an error and indicates
that some code path is not properly setting the tx_Source property.
* "cli" - this string value is set when using roundup-admin and
supplied scripts.
* "web" - this string value is set when using any web based
technique: html interface, xmlrpc ....
* "email" - this string value is set when using an unauthenticated
email based technique.
* "email-sig-openpgp" - this string value is set when email with a
valid pgp signature is used. (*NOTE* the testing for this mode
is incomplete. If you have a pgp infrastructure you should test
and verify that this is properly set.)

This also includes some (possibly incomplete) tests cases for the
modes above and an example of using ts_Source in the customization.txt
document.
files
Fix quote-stripping in mail gateway again. http://hg.code.sf.net:8000/p/roundup/code/#changeset-f1d6e2b2eb640f1379a280ffca401fd9603f4767 Ralf Schlatterbeck rsc@runtux.com 2012-01-05T22:20:11+01:00 2012-01-05T22:20:11+01:00
changeset f1d6e2b2eb64
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix quote-stripping in mail gateway again.
Keep non-quoted first line of quoted section only if it ends in ':'.
files
issue2431638 wrong registration link in bounce mail. http://hg.code.sf.net:8000/p/roundup/code/#changeset-782737d1a2d781ef4248fdb490ceb8a6501ef807 Ralf Schlatterbeck rsc@runtux.com 2012-01-05T21:47:34+01:00 2012-01-05T21:47:34+01:00
changeset 782737d1a2d7
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description issue2431638 wrong registration link in bounce mail.
For non-registered we used a wrong link to the registration page.
files
issue2550535 Fix keep_quoted_text=yes setting. http://hg.code.sf.net:8000/p/roundup/code/#changeset-ba4c632143f61eb8d9ea7beac8e03122bcf8bc91 Ralf Schlatterbeck rsc@runtux.com 2012-01-05T20:55:47+01:00 2012-01-05T20:55:47+01:00
changeset ba4c632143f6
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description issue2550535 Fix keep_quoted_text=yes setting.
In some cases even when keep_quoted_text=yes is configured we would
strip quoted sections. This hit the python bug-tracker especially for
python interpreter examples with leading '>>>' strings. The fix is
slightly different compared to the proposal as this broke
keep_quoted_text=no in certain cases. We also fix a bug where
keep_quoted_text=no would drop the last line of a non-quoted section if
there wasn't an empty line between the next quotes.
files
Be more tolerant when parsing RFC2047 encoded mail headers. http://hg.code.sf.net:8000/p/roundup/code/#changeset-c426cb251bc73362ec1465354c919cecc79145e0 Ralf Schlatterbeck rsc@runtux.com 2012-01-04T18:55:49+01:00 2012-01-04T18:55:49+01:00
changeset c426cb251bc7
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Be more tolerant when parsing RFC2047 encoded mail headers.
Use backported version of my proposed changes to
email.header.decode_header in http://bugs.python.org/issue1079
files
Fix initialisation of crypt option. http://hg.code.sf.net:8000/p/roundup/code/#changeset-8b2ccf6f2cf7c0de1ecf96849c5ff177f4b64a87 Ralf Schlatterbeck rsc@runtux.com 2011-12-16T09:23:24+01:00 2011-12-16T09:23:24+01:00
changeset 8b2ccf6f2cf7
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix initialisation of crypt option.
Bug introduced with gpg support update.
files