http://hg.code.sf.net:8000/p/roundup/code/atom-log/tip/test/test_mailgw.py Mercurial Repository: p/roundup/code: test/test_mailgw.py history 2026-03-18T11:11:03-04:00 test: handle failing email signature removal test differently http://hg.code.sf.net:8000/p/roundup/code/#changeset-67ed90055e476f0f0a8016f2f1d363e13423d512 John Rouillard rouilj@ieee.org 2026-03-18T11:11:03-04:00 2026-03-18T11:11:03-04:00
changeset 67ed90055e47
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description test: handle failing email signature removal test differently

The comment for the test is:

# This fails because the sig isn't removed (we currently remove the
# sig only if the delimiter is the first line in a section)

Originally the test was decorated so it didn't run at all and reported
PASSED. The decorator also returned the value 0. This caused a
deprecation warning:

test/test_mailgw.py::MailgwTestCase::testEmailQuotingRemove3
/usr/lib/python3.12/unittest/case.py:690: DeprecationWarning: It is
deprecated to return a value that is not None from a test case [...]

Better to handle it using pytest xfail so if we start stripping the
signature we get a failure.

This does mean the tests get an xfail registered, but I am ok with
that. I added "OK" to the reason to indicate it was expected.
files
feat: support justhtml parsing library to convert email to plain text http://hg.code.sf.net:8000/p/roundup/code/#changeset-520075b29474aa5dd2586b2e9393ae95b47b0911 John Rouillard rouilj@ieee.org 2025-12-14T22:40:46-05:00 2025-12-14T22:40:46-05:00
changeset 520075b29474
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: support justhtml parsing library to convert email to plain text

justhtml is an pure python, fast, HTML5 compliant parser. It is now an
option for converting html only emails to plain text. Its output
format differs slightly from dehtml or beautifulsoup. Mostly by
removing extra blank lines.

dehtml.py:
Using the stream parser of justhtml. Unable to get the full
document parser to successfully strip script and style blocks.

If I can fix this and use the standard parser, I can in theory
generate markdown from the DOM tree generated by justhtml.

Updated test case to include inline elements that should not cause a
line break when they are encountered. Running dehtml as: `python
roundup/dehtml.py foo.html` will load foo.html and parse it using
all available parsers.

configuration.py: justhtml is available as an option.

docs: updated CHANGES.txt, doc/tracker_config.txt added beautifulsoup
and justhtml to the optional software section of doc/installtion.txt.

test_mailgw.py, .github/workflows/ci-test Updated tests and install
justhtml as part of CI.
files
fix: cae1bbf2536b - expression errors not setting result properly http://hg.code.sf.net:8000/p/roundup/code/#changeset-28c5030757d39722b16ca9923d151d183797bf6f John Rouillard rouilj@ieee.org 2025-01-05T00:59:12-05:00 2025-01-05T00:59:12-05:00
changeset 28c5030757d3
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: cae1bbf2536b - expression errors not setting result properly

Rather than setting 'result' in renderContext, I was wrirting the
output directly, but was stil falling through to where result would be
used.

This caused email to be sent to the admin and when test_mailgw was
called the mail-test.log was somehow created and caused the test there
to fail.

Fixed renderContext to use result variable and output contents of
mail-test.log to make debugging easier.

This should fix CI failure.
files
[permission-performance] Change permission representation http://hg.code.sf.net:8000/p/roundup/code/#changeset-c12377fb4144e63ba869fd0d85ea37b269af1b7a Ralf Schlatterbeck rsc@runtux.com 2024-10-18T16:52:42+02:00 2024-10-18T16:52:42+02:00
changeset c12377fb4144
branch permission-performance
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Change permission representation

Now permissions are checked in different order. Permissions without a
check method (which are cheap to check) are checked first. Only if no
permission is found do we check permissions with check methods.
files
Fix tests http://hg.code.sf.net:8000/p/roundup/code/#changeset-0713c286f71c959ae9dbc37426c0e591b904ed01 John Rouillard rouilj@ieee.org 2023-07-23T22:04:27-04:00 2023-07-23T22:04:27-04:00
changeset 0713c286f71c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix tests

The testEmailBodyUnchangedNewIsYes was using a signature (sig) that
didn't have a blank line before it. Therefore the sig wasn't
recognised as a message section and wasn't a candidate to be removed
even if EMAIL_LEAVE_BODY_UNCHANGED was set to no.

Appended a newline before the signature block so testing with

self.instance.config.EMAIL_LEAVE_BODY_UNCHANGED = 'no'

will cause test to fail and setting it to yes (the test condition)
will cause test to pass by keeping signature.

Make same sig change to testEmailBodyUnchangedFollowupIsYes for same
reason. Add new tests for other permutations:

testEmailBodyUnchangedNewIsNew - keep sig
testEmailBodyUnchangedFollowupIsNew - remove sig
testEmailBodyUnchangedFollowupIsNo - remove sig
files
Add new test new email body unchanged is no. http://hg.code.sf.net:8000/p/roundup/code/#changeset-cde2b45d599a9e3bef5e94b06b3755ea8d531511 John Rouillard rouilj@ieee.org 2023-07-23T20:51:29-04:00 2023-07-23T20:51:29-04:00
changeset cde2b45d599a
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Add new test new email body unchanged is no.

verify that sig is stipped.
files
issue2551262 - make mail gateway subject prefix parsing accept spaces http://hg.code.sf.net:8000/p/roundup/code/#changeset-86862ed039fa659bb2e604f8a5d8329b8ecb3e77 John Rouillard rouilj@ieee.org 2023-03-21T19:46:38-04:00 2023-03-21T19:46:38-04:00
changeset 86862ed039fa
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551262 - make mail gateway subject prefix parsing accept spaces

Allow spaces before/after prefix. Also allow spaces between classname
and id number in prefix designator. So "[ issue 23 ] subject" is
parsed like "[issue23] subject".
files
Changes from review http://hg.code.sf.net:8000/p/roundup/code/#changeset-27c2d7295ba2ff582af9acd9cf06ff05fca21d1e Ralf Schlatterbeck rsc@runtux.com 2022-11-23T21:16:01+01:00 2022-11-23T21:16:01+01:00
changeset 27c2d7295ba2
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Changes from review

mailgw script was still using usage()
files
Add OAuth authentication to the mailgw script http://hg.code.sf.net:8000/p/roundup/code/#changeset-3359dc1dabb09c4594c730feec4eafceb323186f Ralf Schlatterbeck rsc@runtux.com 2022-11-23T10:13:48+01:00 2022-11-23T10:13:48+01:00
changeset 3359dc1dabb0
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add OAuth authentication to the mailgw script

Now IMAPS can be used with OAuth as required by several large cloud
providers. Move command line processing of the mailgw script to
``argparse``. Note that the command line options of the mailgw have
changed, see upgrading.txt for details.
files
issue2551232 - modify in-reply-to threading when multiple matches http://hg.code.sf.net:8000/p/roundup/code/#changeset-bd2c3b2010c33483301a75b52047d52277a5e289 John Rouillard rouilj@ieee.org 2022-09-08T18:49:46-04:00 2022-09-08T18:49:46-04:00
changeset bd2c3b2010c3
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551232 - modify in-reply-to threading when multiple matches

if an email is missing an issue designator, in-reply-to threading is
attempted. In this change if in-reply-to threading matches multiple
issues, fall back to matching on subject. It used to just arbitrairly
choose the first matching issue.
files
Remove pytest deprication warnings. http://hg.code.sf.net:8000/p/roundup/code/#changeset-68f7acd6729831f72ea0b73ebfea0ff1c5106f9b John Rouillard rouilj@ieee.org 2022-06-22T11:51:23-04:00 2022-06-22T11:51:23-04:00
changeset 68f7acd67298
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Remove pytest deprication warnings.

returns of anything but None are depricated for test methods.
files
Add more translation tests for mailgw.py http://hg.code.sf.net:8000/p/roundup/code/#changeset-567283742a5c3b57cce6bc7f27f249dfec74036f John Rouillard rouilj@ieee.org 2022-05-12T14:50:50-04:00 2022-05-12T14:50:50-04:00
changeset 567283742a5c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Add more translation tests for mailgw.py

Verify error in German for [issue999999] and [frobulated] in subject
line. The issue doesn't exist and the class doesn't exist.

Also add warning to mailgw.py for mailgw.py::handle_message that any
chnags to that method need to be replicated for all Translate tests in
test_mailgw.py since it can't be tested directly as it destroys the
mock used for the database by the test harness.
files
Testing translations in mailgw.py, update translations, i18n improved http://hg.code.sf.net:8000/p/roundup/code/#changeset-01216187a16781e67dcd674e0230d79f411442c0 John Rouillard rouilj@ieee.org 2022-05-11T21:20:34-04:00 2022-05-11T21:20:34-04:00
changeset 01216187a167
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Testing translations in mailgw.py, update translations, i18n improved

i18n.py: translation objects get a _file which is the file that catalog
was loaded from. Useful for debugging.

mailgw.py: fix code to include roundup.i18n and invoke propert method.

GNUMakefile: added roundup.pot target. Also building roundup.pot sets
the release version and product code in roundup.pot.
Release version from roundup.__init__.py.

merged roundup.pot changes into all *.po files. Not sure if the
version info in the headers of the .po files is supposed to be
updated to match roundup.pot or not.

test_mailgw.py: test cases for en and de translations tested. Not sure
why I neede to set roundupdb._ and mailgw._, but followed setUp
method code and it worked.
files
Add testing for beautifulsoup4 as html -> text converter. http://hg.code.sf.net:8000/p/roundup/code/#changeset-0ef68871761323e800e6cf62b280a89a3bfc3142 John Rouillard rouilj@ieee.org 2022-05-11T00:52:02-04:00 2022-05-11T00:52:02-04:00
changeset 0ef688717613
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Add testing for beautifulsoup4 as html -> text converter.
files
Start work on supporting testing of beautifulsoup http://hg.code.sf.net:8000/p/roundup/code/#changeset-2453d7b58a92ac058eb4d624122fa439aa6a4e9b John Rouillard rouilj@ieee.org 2022-05-02T17:04:27-04:00 2022-05-02T17:04:27-04:00
changeset 2453d7b58a92
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Start work on supporting testing of beautifulsoup

Change original dehtml only test into a method to be called by a test
method. Converter to be used is passed in and selects the right
comparison text.

All this change does is set things up to run the dehtml test
parameterized. It also provdes the stub for the beautiful soup test.
files
Make memorydb persistent across re-open http://hg.code.sf.net:8000/p/roundup/code/#changeset-7f00fc5958cad7db10bae202b8a22214584fc076 Ralf Schlatterbeck rsc@runtux.com 2021-03-31T12:49:28+02:00 2021-03-31T12:49:28+02:00
changeset 7f00fc5958ca
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Make memorydb persistent across re-open

This allows memorydb to be used for more tests, in particular re-opening
with another user for checking permissions.
files
Fix roundup/test http://hg.code.sf.net:8000/p/roundup/code/#changeset-58817c3bf47111b723b3fca5ef988f5ba9a54f8a Ralf Schlatterbeck rsc@runtux.com 2021-03-30T14:16:28+02:00 2021-03-30T14:16:28+02:00
changeset 58817c3bf471
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix roundup/test

Move the test-detectors in tx_Source_detector.py to roundup/test for two
reasons: It's used in the memorydb convenience functions and it may be
useful in other tests. Make the prefix a paramter of the convenience
functions to be usable in other tests.
files
Fix filename created from mail attachments http://hg.code.sf.net:8000/p/roundup/code/#changeset-14c9284a8bad18fb1cfa50aa7e5c9d54a0522b4b Ralf Schlatterbeck rsc@runtux.com 2021-03-17T10:54:14+01:00 2021-03-17T10:54:14+01:00
changeset 14c9284a8bad
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix filename created from mail attachments

Fixes issue2551118
files
Remove old code import imp, old style trackers db/backend_name http://hg.code.sf.net:8000/p/roundup/code/#changeset-778a9f45506769b8bf5cad1322c3fc3500a4c1d8 John Rouillard rouilj@ieee.org 2020-12-20T15:43:07-05:00 2020-12-20T15:43:07-05:00
changeset 778a9f455067
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Remove old code import imp, old style trackers db/backend_name

Module imp is depricated. Removing that means rewriting old style
trackers that used imp to load schema and config files. So removed
code supporting old style trackers that have been depricated since
2008.

Added test to verify that existence of dbinit.py triggers alert that
tracker is old style and not supported.

Also remove support for depricated db/backend_name file for specifying
backend. It is now specified in config.ini's [rdbms] backend.

It looks like not specifying an [rdbms] backend key in config.ini
throws a config error. However I left in a check and throw an
exception with details if there is an empty backend value. But I don't
think it will ever be triggered.

Removed unused import of imp in a number of test files.
files
Handle auto-submitted header on inbound email like bulk. http://hg.code.sf.net:8000/p/roundup/code/#changeset-ca0915457761df303d6da761287bcf0780dd9b68 John Rouillard rouilj@ieee.org 2020-07-28T20:58:04-04:00 2020-07-28T20:58:04-04:00
changeset ca0915457761
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Handle auto-submitted header on inbound email like bulk.

Part of issue2550564.
files
Removing @pytest.mark.xfail decoration. Ralf fixed the test. http://hg.code.sf.net:8000/p/roundup/code/#changeset-ad0ece9cb01b8e09edb5ae59b13e7743410d0e6d John Rouillard rouilj@ieee.org 2019-07-22T20:20:32-04:00 2019-07-22T20:20:32-04:00
changeset ad0ece9cb01b
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Removing @pytest.mark.xfail decoration. Ralf fixed the test.
files
Fix test for mailer.standard_message http://hg.code.sf.net:8000/p/roundup/code/#changeset-f767a83a6af04e0d7efe5b2b8d36385cce5767b2 Ralf Schlatterbeck rsc@runtux.com 2019-07-22T19:59:28+02:00 2019-07-22T19:59:28+02:00
changeset f767a83a6af0
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix test for mailer.standard_message

Python3 needs normal (unicode) strings while python2 needs UTF-8 encoded
strings for the mail interface.
files
Mark failing test as xfail till it gets fixed. http://hg.code.sf.net:8000/p/roundup/code/#changeset-75586a0c26c04181d38eac644fed7af1c8a856ff John Rouillard rouilj@ieee.org 2019-07-06T09:05:09-04:00 2019-07-06T09:05:09-04:00
changeset 75586a0c26c0
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Mark failing test as xfail till it gets fixed.
files
Add test for standard_message http://hg.code.sf.net:8000/p/roundup/code/#changeset-7c05c481e47001dc0af512ce8d721595388cb764 Ralf Schlatterbeck rsc@runtux.com 2019-07-03T09:03:02+02:00 2019-07-03T09:03:02+02:00
changeset 7c05c481e470
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add test for standard_message

.. with utf-8 subject, TO header, and content.
files
Replace deprecated assertEquals with assertEqual and failUnlessRaises http://hg.code.sf.net:8000/p/roundup/code/#changeset-95a366d460654c841c28b1d7d6ba45cda60b1779 John Rouillard rouilj@ieee.org 2019-06-10T21:27:17-04:00 2019-06-10T21:27:17-04:00
changeset 95a366d46065
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Replace deprecated assertEquals with assertEqual and failUnlessRaises
with assertRaises.
files
issue2551043: Add X-Roundup-issue-id email header. http://hg.code.sf.net:8000/p/roundup/code/#changeset-2c0f89edabe19c562c5f1effefe38d967c869c35 John Rouillard rouilj@ieee.org 2019-06-01T13:19:14-04:00 2019-06-01T13:19:14-04:00
changeset 2c0f89edabe1
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551043: Add X-Roundup-issue-id email header.

Add a new header to make it easier to filter notification emails
without having to parse the subject line. Modified tests to account
for new header.
files
issue2551018 add new nosy_filter parameter to nosymessage. Function http://hg.code.sf.net:8000/p/roundup/code/#changeset-f3d68c1bb96ea8cb96d3bb30b5c773f2886bbb7d John Rouillard rouilj@ieee.org 2019-03-25T19:45:08-04:00 2019-03-25T19:45:08-04:00
changeset f3d68c1bb96e
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551018 add new nosy_filter parameter to nosymessage. Function
passed as nosy_filter can rewrite the nosy message body before it's
sent. Tom Ekberg tekberg did the work.
files
assert_ is depricated. Replacing with assertTrue to reduce logs in travisci. http://hg.code.sf.net:8000/p/roundup/code/#changeset-f8893e1cde0d05513a328bd4698dac10d94805b3 John Rouillard rouilj@ieee.org 2019-03-15T20:29:00-04:00 2019-03-15T20:29:00-04:00
changeset f8893e1cde0d
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description assert_ is depricated. Replacing with assertTrue to reduce logs in travisci.
files
Do not transcode binary email attachments (issue2551004). http://hg.code.sf.net:8000/p/roundup/code/#changeset-081be318661bd712500821e51eb2ae681732570a Joseph Myers jsm@polyomino.org.uk 2018-09-26T01:24:19+00:00 2018-09-26T01:24:19+00:00
changeset 081be318661b
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Do not transcode binary email attachments (issue2551004).

Only transcode attachments if a charset is specified or if they have a
text/* content-type.
files
Mark failing test http://hg.code.sf.net:8000/p/roundup/code/#changeset-52e50ab5dfda31e1573a80bc9260d0adfcffb58c Ralf Schlatterbeck rsc@runtux.com 2018-09-25T17:06:18+02:00 2018-09-25T17:06:18+02:00
changeset 52e50ab5dfda
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Mark failing test
files
Add failing transcoding test http://hg.code.sf.net:8000/p/roundup/code/#changeset-4523fe3cf04cd1b920afbc06bc0e24134ef0673d Ralf Schlatterbeck rsc@runtux.com 2018-09-25T17:03:19+02:00 2018-09-25T17:03:19+02:00
changeset 4523fe3cf04c
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add failing transcoding test

An attachment with Content-Type application/octet is transcoded to
utf-8, originally the attachment is in latin1 (although that doen't
matter, no charset can be asumed for a binary-only attachment).
files
Fix email interfaces with Python 3 (issue 2550974, issue 2551000). http://hg.code.sf.net:8000/p/roundup/code/#changeset-29346d92d80cdfb50a30186dc1e45414dcdecf56 Joseph Myers jsm@polyomino.org.uk 2018-09-16T13:55:53+00:00 2018-09-16T13:55:53+00:00
changeset 29346d92d80c
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Fix email interfaces with Python 3 (issue 2550974, issue 2551000).

This patch fixes various issues handling incoming email with
roundup-mailgw with Python 3.

Incoming email must always be handled as bytes, not strings, because
it may contain 8-bit-encoded MIME parts with different encodings in
each part. When handling piped input, that means using
sys.stdin.buffer in Python 3 for binary input, along with
message_from_binary_file, not sys.stdin which is text input and may be
for the wrong encoding and not message_from_file. (In turn, tests
that use MailGW.main with text input are affected so an s2b call is
inserted in the test code and it is made to use BytesIO not StringIO.
Properly all the test messages in test_mailgw.py ought to use b''
explicitly rather than having such an s2b conversion, and there ought
to be test messages using 8-bit encodings with non-ASCII characters to
verify that that case works.)

imaplib and poplib return bytes not strings with Python 3 (from
inspection of the code, not tested), as is necessary for the above
reasons. Thus, the handling of IMAP and POP messages must expect
bytes and handle the data accordingly.

For messages from mailboxes, I saw the same problem described in issue
2551000 for a multipart message with a single (non-ASCII) part. The
Roundup code requires RoundupMessage not email.message.Message to be
used recursively for all MIME parts of a message. Because the mailbox
module uses email.message_from_* directly without passing the _class
argument to them, fixing this requires temporarily patching the email
module to ensure _class=RoundupMessage gets passed to those methods.
files
be more lenient when comparing string results http://hg.code.sf.net:8000/p/roundup/code/#changeset-19bd4b413ed6c9da8233601afeec34540486e5cf Christof Meerwald cmeerw@cmeerw.org 2018-08-18T23:04:22+01:00 2018-08-18T23:04:22+01:00
changeset 19bd4b413ed6
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description be more lenient when comparing string results
files
compare folded headers in mailgw tests http://hg.code.sf.net:8000/p/roundup/code/#changeset-497ab896397d3043bbf824ed348d795b6bcdbdf9 Christof Meerwald cmeerw@cmeerw.org 2018-08-16T19:23:14+01:00 2018-08-16T19:23:14+01:00
changeset 497ab896397d
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description compare folded headers in mailgw tests
files
use gpg module instead of pyme module for PGP encryption http://hg.code.sf.net:8000/p/roundup/code/#changeset-b7fa56ced601c5293f7a5d735eaf193973a2f682 Christof Meerwald cmeerw@cmeerw.org 2018-08-12T16:17:14+01:00 2018-08-12T16:17:14+01:00
changeset b7fa56ced601
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description use gpg module instead of pyme module for PGP encryption
files
updated mailgw to no longer use mimetools based on jerrykan's patch http://hg.code.sf.net:8000/p/roundup/code/#changeset-725266c03eab7b5df86fa532c40123826ee7fc2b Christof Meerwald cmeerw@cmeerw.org 2018-08-12T16:15:10+01:00 2018-08-12T16:15:10+01:00
changeset 725266c03eab
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description updated mailgw to no longer use mimetools based on jerrykan's patch
files
encoding fixes in test cases http://hg.code.sf.net:8000/p/roundup/code/#changeset-b0048969990d2e38b35fd7cbc56408c15db1a975 Christof Meerwald cmeerw@cmeerw.org 2018-07-29T02:45:39+01:00 2018-07-29T02:45:39+01:00
changeset b0048969990d
branch
bookmark
tag
user Christof Meerwald <cmeerw@cmeerw.org>
description encoding fixes in test cases
files
Python 3 preparation: StringIO. http://hg.code.sf.net:8000/p/roundup/code/#changeset-55f09ca366c484e1c06dd34a4efead3f1634d7d7 Joseph Myers jsm@polyomino.org.uk 2018-07-25T09:08:29+00:00 2018-07-25T09:08:29+00:00
changeset 55f09ca366c4
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: StringIO.

This generally arranges for StringIO and cStringIO references to use
io.StringIO for Python 3 but io.BytesIO for Python 2, consistent with
the string representations generally used in Roundup. A special
FasterStringIO in the TAL code, which referenced internals of the old
Python 2 StringIO module, is cut down so it doesn't actually do
anything beyond the StringIO class it inherits from (it would also be
reasonable to remove FasterStringIO completely). One place in
roundup_server.py clearly needing binary I/O is made to use io.BytesIO
unconditionally.
files
Python 3 preparation: make relative imports explicit. http://hg.code.sf.net:8000/p/roundup/code/#changeset-d26921b851c328681aa6638475ce2cdae29963b4 Joseph Myers jsm@polyomino.org.uk 2018-07-24T22:22:08+00:00 2018-07-24T22:22:08+00:00
changeset d26921b851c3
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: make relative imports explicit.

Tool-generated patch.
files
Python 3 preparation: change "x.has_key(y)" to "y in x". http://hg.code.sf.net:8000/p/roundup/code/#changeset-0942fe89e82e3e84d95a3d0ca5d8d15502f57348 Joseph Myers jsm@polyomino.org.uk 2018-07-24T22:08:17+00:00 2018-07-24T22:08:17+00:00
changeset 0942fe89e82e
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: change "x.has_key(y)" to "y in x".

(Also likewise "not in" where appropriate.) Tool-generated patch.
files
Python 3 preparation: "raise" syntax. http://hg.code.sf.net:8000/p/roundup/code/#changeset-35ea9b1efc1494f29fb07690e7087686de58e0f3 Joseph Myers jsm@polyomino.org.uk 2018-07-24T21:39:58+00:00 2018-07-24T21:39:58+00:00
changeset 35ea9b1efc14
branch
bookmark
tag
user Joseph Myers <jsm@polyomino.org.uk>
description Python 3 preparation: "raise" syntax.

Changing "raise Exception, value" to "raise Exception(value)".
Tool-assisted patch. Particular cases to check carefully are the one
place in frontends/ZRoundup/ZRoundup.py where a string exception
needed to be fixed, and the one in roundup/cgi/client.py involving
raising an exception with a traceback (requires three-argument form of
raise in Python 2, which as I understand it requires exec() to avoid a
Python 3 syntax error).
files
issue2550957: Duplicate emails (with patch). http://hg.code.sf.net:8000/p/roundup/code/#changeset-a4fe76fba54a695fd3c31322da662875f4889a09 John Rouillard rouilj@ieee.org 2017-11-07T20:58:48-05:00 2017-11-07T20:58:48-05:00
changeset a4fe76fba54a
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550957: Duplicate emails (with patch).

Bcc and cc users passed to nosymessage are not properly recorded.
This results in duplicate emails. Patch by Trent Gamblin (trentgg).
Applied by and test added by John Rouillard.
files
issue2550799: provide basic support for handling html only emails http://hg.code.sf.net:8000/p/roundup/code/#changeset-e20f472fde7d9ed7e1e5120de2941181efdaeba7 John Rouillard rouilj@ieee.org 2017-10-13T21:46:59-04:00 2017-10-13T21:46:59-04:00
changeset e20f472fde7d
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550799: provide basic support for handling html only emails

Initial implementation and testing with the dehtml html converter
done.

The use of beautifulsoup 4 is not tested. My test system breaks when
running dehtml.py using beautiful soup. I don't get the failures when
running under the test harness, but the text output is significantly
different (different line breaks, number of newlines etc.)

The tests for dehtml need to be generated for beautiful soup and the
expected output changed. Since I have a wonky install of beautiful
soup, I don't trust my output as the standard to test against. Also
since beautiful soup is optional, the test harness needs to skip the
beautifulsoup tests if import bs4 fails. Again something outside of my
expertise. I deleted the work I had done to implement that. I could
not get it working and wanted to get this feature in in some form.
files
Use Python-3-compatible 'as' syntax for except statements http://hg.code.sf.net:8000/p/roundup/code/#changeset-198b6e810c67c4e7e38f1aa41edbc864783fe6e6 Eric S. Raymond esr@thyrsus.com 2017-08-24T22:21:37-04:00 2017-08-24T22:21:37-04:00
changeset 198b6e810c67
branch
bookmark
tag
user Eric S. Raymond <esr@thyrsus.com>
description Use Python-3-compatible 'as' syntax for except statements

Many raise statements near these are also fixed.
So are two ivorrect file encoding marks ('utf8'->'utf-8').
files
partly test subject line of 'help' to trigger help message. Can't verify the help message is sent though http://hg.code.sf.net:8000/p/roundup/code/#changeset-65b42fd96c009dc1b742ad9190391e0adbafa459 John Rouillard rouilj@ieee.org 2016-07-20T21:35:55-04:00 2016-07-20T21:35:55-04:00
changeset 65b42fd96c00
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description partly test subject line of 'help' to trigger help message. Can't verify the help message is sent though
files
Fix subject parsing in mail gateway. http://hg.code.sf.net:8000/p/roundup/code/#changeset-6bbb6dd974589a52c5f1063c421b6456260fc0ba Ralf Schlatterbeck rsc@runtux.com 2016-07-19T11:47:53+02:00 2016-07-19T11:47:53+02:00
changeset 6bbb6dd97458
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Fix subject parsing in mail gateway.

The previous parsing routine would not ensure that arguments are at the
end of the subject and when subject_suffix_parsing was configured to be
'loose' it would truncate the subject when encountering a double prefix,
e.g. Subject: [frobulated] [frobulatedagain] this part would be lost
files
Add test for msg_header_property of assignedto http://hg.code.sf.net:8000/p/roundup/code/#changeset-0c9eed9c40293f5bc86ab1e5da11579f22c3f801 Ralf Schlatterbeck rsc@runtux.com 2016-07-18T13:55:34+02:00 2016-07-18T13:55:34+02:00
changeset 0c9eed9c4029
branch
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Add test for msg_header_property of assignedto
files
adding test for issue934009. Code coverage shows no test for MAILGW_LEAVE_BODY_UNCHANGED=yes. Adding that for new issue creation and followup. http://hg.code.sf.net:8000/p/roundup/code/#changeset-410e565b5c1fc290edf2b667efe4719d8272aeb6 John Rouillard rouilj@ieee.org 2016-07-03T20:44:26-04:00 2016-07-03T20:44:26-04:00
changeset 410e565b5c1f
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description adding test for issue934009. Code coverage shows no test for MAILGW_LEAVE_BODY_UNCHANGED=yes. Adding that for new issue creation and followup.
files
Fix issue2550751: Email Header Issue. http://hg.code.sf.net:8000/p/roundup/code/#changeset-a927f9549af0f703ddfe8ee10ef4cb02dd7b6a67 John Rouillard rouilj@ieee.org 2016-07-03T19:29:50-04:00 2016-07-03T19:29:50-04:00
changeset a927f9549af0
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix issue2550751: Email Header Issue.

Noel Garces requested the ability to suppress email headers like
"x-roundup-issue-files". With Ralf's addition of the Link/Multilink
property attribute 'msg_header_property' we can do this
easily. Setting the 'msg_header_property' to the empty string '' (not
to None) will suppress the header for that property.

3 line code change, single test, doc updated and change note.
files
Fixed incorrect header comparisons in compareMessages. It iterated http://hg.code.sf.net:8000/p/roundup/code/#changeset-226052e0cc4cd2d289467238203fffd9549759fb John Rouillard rouilj@ieee.org 2016-07-03T18:54:18-04:00 2016-07-03T18:54:18-04:00
changeset 226052e0cc4c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fixed incorrect header comparisons in compareMessages. It iterated
over every header in the new message and didn't detect when the new
message was missing a header that was in the reference message.

I have fixed that by retrieving all header names from the new and
reference (old) messages, lower casing the list and comparing them
(with a fixup for the x-roundup-version header). If the list of
headers isn't identical, I throw an error.

Also the Content-Type mime values are now being compared as well to
make sure the new message being generates matches the reference mime
content type.

Of course this broke some tests. A number of them were using
compareMessages when they should have been using assertEqual. That was
an easy fix.

Also some messages when retrieved from the content property are
missing trailing newlines. I think this is part of the signature
removal algorithm. This may be a bug or intended. In any case I am
fixing the tests to remove trailing newlines.

However I have a handful of tests that are not that obvious.
Two of are

test_mailgw.py:testMultipartCharsetLatin1AttachFile
test_mailgw.py:testMultipartCharsetUTF8AttachFile

I removed a:

Content-Transfer-Encoding: quoted-printable

message header from the reference message in both cases. The message
content-type was multipart/mixed and it had no content that was
outside of a mime part. Sending a message like this using mailx and
nmh resulted in an email being delivered that was missing a
Content-Transfer-Encoding, so I think I am ok here.

The last broken test also started as a missing
Content-Transfer-Encoding header. But there was a twist:

test-mailgw.py:testMultipartRFC822

had a reference copy with a mime type of text/plain which requires a
Content-Transfer-Encoding. However I never looked at the mime type of
the mail being generated. The generated mail was of type
multipart/mixed. So I changed the reference copy to remove the
Content-Transfer-Encoding and changed the reference mime type to
multipart/mixed.

Now all the mailgw tests are passing. With luck I haven't encoded an
invalid test.

Also did one minor spelling correction.
files
Fix issue934009: Have New Issues Submitted By Email *Not* Change Body! http://hg.code.sf.net:8000/p/roundup/code/#changeset-14abd0a6720774675b58f9e4e246c17dc96e49e7 John Rouillard rouilj@ieee.org 2016-07-02T20:05:06-04:00 2016-07-02T20:05:06-04:00
changeset 14abd0a67207
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix issue934009: Have New Issues Submitted By Email *Not* Change Body!

The mailgw config options: keep_quoted_text and leave_body_unchanged
can now have a new values: new. If set to new, keep_quoted_text acts
like yes if the message is starting a new issue. Otherise it strips
quoted text.

This allows somebody to start a new issue by forwarding
a threaded email (with multiple quoted parts) into roundup and
keeping all the quoted parts. If leave_body_unchanged is set to
new, even the signature on the email that starts a new issue will be
preserved.
files
Improved work-around for pytest markers bug http://hg.code.sf.net:8000/p/roundup/code/#changeset-43a1f7fe39f502d3dc1ba5c560a81a629c924619 John Kristensen john@jerrykan.com 2016-06-28T15:39:38+10:00 2016-06-28T15:39:38+10:00
changeset 43a1f7fe39f5
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Improved work-around for pytest markers bug

The previous fix was only a partial solution. Any test class sharing a
parent with, and appearing after, a skipped test class was being skipped
(not just other test classes using the skip/skipif marker). Now only
tests that should be skipped will be skipped, the rest should run as
normal.
files
Fix work-around for pytest markers bug http://hg.code.sf.net:8000/p/roundup/code/#changeset-37d1e24fb941044d39cfdd59449155eeee154b65 John Kristensen john@jerrykan.com 2016-06-27T14:03:32+10:00 2016-06-27T14:03:32+10:00
changeset 37d1e24fb941
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Fix work-around for pytest markers bug

The initial work-around implemented was totally botched using
'pytest.skip' instead of 'pytest.mark.skip' which resulted in all tests
in a file being completely ignored if any skip conditions that evaluated
to true were declared or imported in the file.

This work-around will not correctly display why all the tests have been
skipped when using the '-rs' parameter. Only the first skip marker to
taint a parent test class will be displayed (ie. if both xapian and
mysql tests are being skipped, pytest will only output that tests are
being skipped because xapian is not installed even though the mysql
tests are also being skipped because mysql backend is not available).

There also seems to be a bug in the current version of pytest being used
in 'run_tests.py' (v2.8.4) that results in the skip not actually working
when using 'pytest.mark.skip'. This does work correctly with the most
recent release (v2.9.2), so the 'run_tests.py' script will need to be
updated.
files
issue2550803: Replying to NOSY mail goes to the tracker through http://hg.code.sf.net:8000/p/roundup/code/#changeset-99e28935979856b69d212a9da77491d4acaf04cb John Rouillard rouilj@ieee.org 2016-06-26T00:36:23-04:00 2016-06-26T00:36:23-04:00
changeset 99e289359798
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550803: Replying to NOSY mail goes to the tracker through
reply-to, not original message author.

Created new [tracker] replyto_address config.ini option to allow:
1) setting reply-to header to the tracker
2) setting reply-to header to the address of the author
of the change
3) setting it to a fixed address (like noreply@some.place)

Proposal by Peter Funk (pefu) in discussion with Tom Ekberg
(tekberg).

I chose to re-retrieve the email address for the author from the
database rather than adding a new variable. Also managed to make
a test case for each of the three settings.
files
issue2109308 add subject argument to nosymessage. Initial patch Frank Niessink. Test and application with slight mods by rouilj http://hg.code.sf.net:8000/p/roundup/code/#changeset-786f0581bc90fad922578e311bcf861d9535dc90 John Rouillard rouilj@ieee.org 2016-06-11T18:06:18-04:00 2016-06-11T18:06:18-04:00
changeset 786f0581bc90
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2109308 add subject argument to nosymessage. Initial patch Frank Niessink. Test and application with slight mods by rouilj
files
issue2550763 Strip whitespace from Multilink values after + or -. http://hg.code.sf.net:8000/p/roundup/code/#changeset-2706b66675bf32645c91eb7eeb3255d6b2ff7043 John Rouillard rouilj@ieee.org 2016-06-05T20:48:07-04:00 2016-06-05T20:48:07-04:00
changeset 2706b66675bf
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550763 Strip whitespace from Multilink values after + or -.

Diagnosis, patch developed by W. Trevor King. Test heavily modified by
John Rouillard.

The goal is to make:

Subject: [issue23455] ... [ keyword =+ Foo,Foo Zot ; nosy =+ mary ]

act like:

Subject: [issue23455] ... [ keyword =+Foo,Foo Zot; nosy =+mary]

Without the patch " Foo" is not recognized.
files
Fix issue1615201: Added a new configuration option EMAIL_KEEP_REAL_FROM http://hg.code.sf.net:8000/p/roundup/code/#changeset-a46d5d0fd5f83d3ef46be378f76cacce3feca681 Peter Funk pf@artcom-gmbh.de 2016-03-18T16:18:20+01:00 2016-03-18T16:18:20+01:00
changeset a46d5d0fd5f8
branch
bookmark
tag
user Peter Funk <pf@artcom-gmbh.de>
description Fix issue1615201: Added a new configuration option EMAIL_KEEP_REAL_FROM
which makes the mail gateway ignore a Resent-From:-header and use the
real From:-header of the original author for further processing of the
message. Setting this option to 'yes' restores the original behaviour
of Roundup before the change in version 0.7.0 where the processing of
Resent-From: was added unconditionally.
files
Work-around for pytest.mark.skipif() bug http://hg.code.sf.net:8000/p/roundup/code/#changeset-c977f3530944c9adf85594092b8caff9c13889fd John Kristensen john@jerrykan.com 2015-09-07T23:24:33+10:00 2015-09-07T23:24:33+10:00
changeset c977f3530944
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Work-around for pytest.mark.skipif() bug

There is a pytest bug that can cause all test classes marked with the
skipif() decorator that inherit a common class to be skipped if one of
the skipif() conditions is True. See:

https://github.com/pytest-dev/pytest/issues/568
files
Remove unneeded TestSuite code from tests http://hg.code.sf.net:8000/p/roundup/code/#changeset-364c549918613cff81c3ebe07aee4025e0ef3fcc John Kristensen john@jerrykan.com 2015-08-21T13:08:02+10:00 2015-08-21T13:08:02+10:00
changeset 364c54991861
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Remove unneeded TestSuite code from tests

The TestSuite code is no longer needed now that we are using py.test
which can automatically discover tests
files
Replace existing run_tests.py script with a pytest script http://hg.code.sf.net:8000/p/roundup/code/#changeset-380d8d8b30a3b0dca517fb242809703963a00588 John Kristensen john@jerrykan.com 2015-08-20T18:15:23+10:00 2015-08-20T18:15:23+10:00
changeset 380d8d8b30a3
branch
bookmark
tag
user John Kristensen <john@jerrykan.com>
description Replace existing run_tests.py script with a pytest script

The existing run_test.py script is quite old, a bit restrictive, and
doesn't always behave as documented. The pytest testing tool is mature,
well documented, and maintained.

The run_tests.py script is generating by installing py.test and running:

py.tests --genscript=run_tests.py

Note: to generate a script that is compatible with python2.6 the command
needs to be run using python2.6
files