Mercurial > p > roundup > code
changeset 8071:a4cb4e75d4e9
final changes for 2.4.0 release
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Fri, 12 Jul 2024 20:32:20 -0400 |
| parents | 34d5c1303a2e |
| children | 3a2762144b65 |
| files | CHANGES.txt MANIFEST.in doc/announcement.txt doc/installation.txt doc/upgrading.txt website/www/index.txt |
| diffstat | 6 files changed, 349 insertions(+), 70 deletions(-) [+] |
line wrap: on
line diff
--- a/CHANGES.txt Tue Jul 09 22:13:25 2024 -0400 +++ b/CHANGES.txt Fri Jul 12 20:32:20 2024 -0400 @@ -11,6 +11,7 @@ v2.7.2 or later are required to run newer releases of Roundup. Roundup 2.0 supports Python 3.4 and later. Roundup 2.1.0 supports python 3.6 or newer (3.4/3.5 might work, but they are not tested). +Roundup 2.4.0 is the last release to support Python 2. 2024-07-13 2.4.0 @@ -257,7 +258,7 @@ your network using 'roundup-demo ... -B hostname -H hostname'. (John Rouillard) - issue2551347 - make _generic.help.html work without property - settings. THis applies to classic or minimal trackers. It allows use + settings. This applies to classic or minimal trackers. It allows use of classhelp without the property seting for informtion only (e.g. description of what a priority or status means) without being able to select the property in the classhelper. Good for adding help
--- a/MANIFEST.in Tue Jul 09 22:13:25 2024 -0400 +++ b/MANIFEST.in Fri Jul 12 20:32:20 2024 -0400 @@ -11,4 +11,5 @@ include demo.py *.txt global-exclude *.pyc *.pyo .DS_Store *.orig *.rej *~ \#* .flake8 include doc/conf.py doc/Makefile doc/requirements.pip doc/roundup-favicon.ico +include doc/format_config.awk include locale/*.po locale/*.mo locale/roundup.pot locale/Makefile
--- a/doc/announcement.txt Tue Jul 09 22:13:25 2024 -0400 +++ b/doc/announcement.txt Fri Jul 12 20:32:20 2024 -0400 @@ -1,12 +1,16 @@ I'm proud to release version 2.4.0 of the Roundup issue -tracker. This release is a bugfix and feature -release, so make sure to read `docs/upgrading.txt +tracker. This release is a bugfix and feature release, so +make sure to read `docs/upgrading.txt <https://www.roundup-tracker.org/docs/upgrading.html>`_ to bring your tracker up to date. The 79 changes, as usual, include some new features and many bug fixes. +Version 2.4.0 will be the last release to support Python +2. The next minor release, planned for mid 2025, will occur +5 years after Roundup started supporting Python 3. + Note that you should run ``roundup-admin ... migrate`` to update the database schema version. Do this before you use the web, command-line or mail interface and before any users @@ -91,9 +95,9 @@ * sqlite version 1 and StructuredText support removed. The file CHANGES.txt has a detailed list of feature -additions and bug fixes for each release. The most -recent changes from there are at the end of this -announcement. Also see the information in doc/upgrading.txt. +additions and bug fixes for each release. The most recent +changes from there are at the end of this announcement. Also +see the information in doc/upgrading.txt. If you find bugs, please report them to issues AT roundup-tracker.org or create an account at @@ -113,13 +117,12 @@ this before you use the web, command-line or mail interface and before any users access the tracker. -Roundup requires Python 2 newer than version 2.7.12 or -Python 3 newer than or equal to version 3.6 for correct -operation. (Python 3.4 or 3.5 may work, but are not tested.) -Note that Python 2 support is being removed from the CI -platforms, so you should deploy new trackers with Python 3 -and plan on upgrading older trackers from Python 2 to Python -3. See the upgrade guide. +Roundup requires Python 2 newer than version 2.7.12 or Python 3 +newer than or equal to version 3.6 for correct operation. (Python +3.4 or 3.5 may work, but are not tested.) Note that Roundup 2.4.0 +will be the last release to support Python 2. You should deploy +new trackers with Python 3 and plan on upgrading older trackers +from Python 2 to Python 3. See the upgrade guide. To give Roundup a try, just download (directions above), unpack and run:: @@ -181,4 +184,268 @@ From 2.3.0 to 2.4.0 -TBA +Fixed: + +- CVE-2024-39124 - The classhelpers (_generic.help.html) are + vulnerable to an XSS attack. A specially crafted URL that used + that endpoint would result in running a script embedded in the + URL. (Found/reported by Alec Romano (4rdr), fix/tests John + Rouillard) +- CVE-2024-39125 - If the Referer header is set to a script tag, + it will be executed when the error in the Referer header is + reported. (Found/reported by Alec Romano (4rdr), fix/tests John + Rouillard) +- CVE-2024-39126 - PDF, XML and SVG files attached to an issue can contain + embedded JavaScript. This JavaScript was executed when the file was + accessed. PDF files are now downloaded and not displayed in the + browser. A content security policy is added for all download files + which prevents code execution in SVG files. (Found/reported by Alec + Romano (4rdr), fix/tests John Rouillard) +- issue2551282 - MySQL utf8mb4 issues and + issue2551115 - Use utf8mb4 as a default for MySQL instead of utf8 + The default database type and collations have been set to: + utf8mb4, utf8mb4_unicode_ci and utf8mb4_0900_bin. They are (sadly) + configurable from config.ini. Require directions on upgrading the + MySQL db have been documented in upgrading.txt. +- issue2551063 - Rest/Xmlrpc interfaces needs failed login protection. + Failed API login rate limiting with expiring lockout added. (John + Rouillard) +- issue2551184 - improve i18n handling. Patch to test to make sure it + uses the test tracker's locale files and not other locale + files. (Marcus Priesch) +- issue2551283 - fail if version 2.4.9 of markdown2 is used, it broke + [issue1](issue1) style links. Support markdown2 2.4.8 and earlier + and 2.4.10 with its new schema filtering method. (John Rouillard) +- multiple flake8 fixes (John Rouillard) +- rename loop variable in 'for sendto in sendto:' (John Rouillard) +- issue2551193 - Fix roundup for removal of cgi and cgitb standard + python modules (and FieldStorage/MiniFieldStorage). Replaced imports + from cgi to use roundup.anypy.cgi_ which will load the system cgi + unless it is missing. Then it will load roundup.anypy.vendored.cgi + and make *FieldStroage symbols available. Roundp uses its own + cgitb.py and not the system cgitb.py. It looks like it's the + precursor to the system cgitb.py. (John Rouillard) +- issue2551278 - datetime.datetime.utcnow deprecation. Replace + calls with equivalent that produces timezone aware dates rather than + naive dates. (John Rouillard) +- when using "roundup-admin display" indent the listing only if + headers or protected fields are requested. This makes the output + look like it did previously to 2.3.0 if the new features aren't + used. Roundup-admin output was never meant to be machine parsed, but + don't break it unless required. (John Rouillard) +- issue2551290 - pip install roundup Hangs on Windows 10 + The install under windows goes into an infinite loop using pip or + source install. (John Rouillard) +- Document use of pyreadline3 to allow roundup-admin to have CLI editing + on windows. (John Rouillard) +- issue2551293 - remove schema_hook from Tracker instance. Looks like + it was an obsolete hook used for testing. Never documented and not + accessible from schema.py. +- Fix roundup-admin security command. Lowercase its optional + argument. Roles are indexed by lower case role name. So 'security + User' and 'security user' should generate the same output. (John + Rouillard from issue on mailing list by Chuck Cunningham) +- make roundup-server exit more quickly on ^C. This seems to be + limited to windows. (John Rouillard) +- Fix error handling so failure during import of a non-user item + doesn't cause a second traceback. (Found by Norbert Schlemmer, fix + John Rouillard) +- Handle out of memory error when importing large trackers in + PostgreSQL. (Found by Norbert Schlemmer, extensive testing by + Norbert, fix John Rouillard) +- use unittest.mock rather than mock for + test/test_hyperdbvals.py. (found by Ralf Schlatterbeck. Fix John + Rouillard) +- disable proxy with wget in roundup_healthcheck. (Norbert Schlemmer + Noschvie on github.com) +- support dicttoxml2.py for Roundup running on 3.7 and + newer. dicttoxml uses a type alias: collection.Iterator that is + dropped in Python 3.10. (found by Norbert Schlemmer, fix John + Rouillard) +- fix duplicate html id 'password' in user.item.html in all templates except + jinja2. (John Rouillard) +- fix unclosed file when saving index in indexer_dbm.py. (John Rouillard) +- fix task index in devel tracker so it doesn't cause a crash if all + fields are selected. (John Rouillard) +- fix windows install. When using pip share directory is installed in + a directory tree under the lib directory. Fix it so that Lib/share + is used to install the share tree. The lets Roundup find tracker + templates and translation files. (Found by Simon Eigeldinger, fix + John Rouillard) +- fix roundup-demo, interactive mode would nuke an existing tracker. + (Found Tonu Mikk, fix John Rouillard) +- fix detection/reporting when using a SQLite3 library without FTS5 + support. Install docs updated to state that FTS5 support is required + when using SQLite for back end. (Found Tonu Mikk, fix John + Rouillard) +- issue2551320: user.help-search.html doesn't respect + properties. Setting url parameter properties when using the + classhelp for users now shows the requested properties. (Found by + Patel Malav and Nikunj Thakkar of the UMass-Boston CS682 Spring + 2024 class; fix John Rouillard) +- use ast.eval_literal() rather than eval() to turn CSV exported + string values into Python object/values. +- use template's guess at Content-Type in headers only if Content-Type + is not already set. This allows a template to set its own content + type. For example: _generic.translate can set content type (via + request.client.additional_headers) to application/json and return + json from the template. This json could access the 1i18n functions + for a javascript helper. (John Rouillard) +- when template processing raises an exception the line number is + sometimes missing. This causes cgitb to raise a second exception + which clobbers the info about the template issue. As a stop-gap set + the line number to -1 so the original traceback can be seen. This + could be a bug in ZopeTAL. (John Rouillard) +- issue2551328 - REST results show next link if number of results is a + multiple of page size. There should be no next link. (Found by Patel + Malav and Bharath Kanama of the UMass-Boston CS682 Spring 2024 + class; fix John Rouillard) +- issue2551264 - REST X-Total-Count header and @total_size count + incorrect when paginated - correct values are now returned. + (John Rouillard) +- issue2551331 - Fix repeat first/last methods. (John Rouillard) +- Fix import/export on windows. Use unix line terminating characters. + (John Rouillard) +- Fix anydbm session/otks clear() method on windows when backed by + dumbdbm. Also make anydbm detect the initialized database when + using dumbdbm. (John Rouillard) +- Use of '-' directory in static_files config option under windows + Python fixed. (John Rouillard) +- issue2551334 - number of test bugs that prevented test suite from + running under Windows Python are fixed. WIP. (John Rouillard) +- issue2551302 - Remove support for sqlite version 1 from + back_sqlite.py. We have been using sqlite3 for over a decade. (John + Rouillard) +- issue2551285 - Remove StructuredText support. reStructuredText is + still supported. (John Rouillard) +- Use roundup-demo -p option to set listening port. Was ignored + before. (John Rouillard) +- issue2551346 - Classic tracker's statusauditor raises error if + detectors/config.ini missing + STATUSAUDITOR_CHATTING_REQUIRES_TWO_USERS. The statusauditor.py for + jinja2 and classic templates has been changed to assume that this + option is off when the setting is missing from + detectors/config.ini. Other templates do not implement this option. + (John Rouillard) +- issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for + cgitb.py crash due to pydoc.html.header() signature change. (Patch + by Andrew (kragacles), applied John Rouillard) +- issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for + mailer.py crash due to change in starttls signature change. (Patch + by Andrew (kragacles), modified and applied John Rouillard) +- make classhelper link open in a new window by setting + target="_blank". This prevents overwriting of current page with the + classhelper if javascript is disabled. (John Rouillard) +- issue2551341 - if @columns missing from an index url, the + group headers colspan property = 0. Add "or 100" in + stanza's so headers span all rows (up to 100). +- fix roundup-server response requiring a 301 redirect. Did + not set content length leading to hang/error. (John + Rouillard) +- report basename of filename when template file is invalid + rather than reporting a TypeError. (John Rouillard) +- Make Last-Modified header use GMT not -0000 timezone. Fix error + reported by redbot testing. (John Rouillard) +- Send Vary: Accept-Encoding on any file that could be compressed + even if the file is not encoded/compressed. Found by Redbot + testing. (John Rouillard) +- make If-None-Match work for static file (@@file) case. Found by + Redbot testing (John Rouillard) +- Send vary: accept-encoding for if-modified-since conditional + requests where the file is not modified. (John Rouillard) +- Update JWT example in rest.py to use replacement for + datetime.datetime.utcnow(). (John Rouillard) +- issue2551219 - document requirements of PEM file when using + roundup-server in SSL/TLS mode. Report better error messages + when PEM file is missing certificate or private key. (John + Rouillard) +- Cleanup tracker index generation by roundup-server. Send + correct Content-Length headers so HTTP/1.1 connections don't + hang. (John Rouillard) +- Fix delay when using csv export actions. The CSV file is written + incrementally, so we can't determine the Content-Length. When using + HTTP/1.1, this causes a delay while the browser waits for a timeout. + Forcing the connection to close after the CSV file is written + removes the delay. (John Rouillard) + +Features: + +- issue2551323 - Remove XHTML support. Disabled option to set + html_version to xhtml. Running roundup commands with html_version + set to xhtml will result in an "Invalid value for HTML_VERSION: + 'xhtml'" error. (John Rouillard) +- issue2551103 - add pragma 'display_protected' to roundup-admin. If + true, print protected attributes like id, activity, actor... + when using display or specification subcommands. (John Rouillard) +- add -P pragma=value command line option to roundup-admin. Allows + setting pragmas when using non-interactive mode. (John Rouillard) +- issue685275 - add pragma show_retired to control display of retired + items when using list/table. Add pragma display_header to print + headers for display command. Header displays designator and + retired/active status. (John Rouillard) +- issue2551299 - support config.ini rdbms option 'service'. Allow use + of a PostgreSQL connection service file (pg_service.conf) for + configuring database on a per-tracker basis. Also replaces use of + PGSERVICE env variable for single instance trackers. (From ML + question by ivanov. John Rouillard) +- issue2550852 - support for specifying a PostgreSQL schema to use for + the Roundup database. (Patch by Stuart McGraw; slight modifications, + tests, docs: John Rouillard). +- issue2551274: add configurable logging for REST API when something + fails, we now log status code and error message. + (Ralf Schlatterbeck) +- issue2551317 - add some Jinja2 examples to customizing.txt + document. (John Rouillard) +- multiple scripts/... updates - Python3, linting, enhancements: + weekly-report,schema-dump.py, roundup-reminder, copy-user.py, + dump_dbm_sessions_db.py, contributors.py (John Rouillard) +- roundup/msgfile.py can now be called as 'python msgfmt.py de.po de.mo' + or 'python msgfmt.py -o de.mo de.po' to compile a translation file if + GNU msgfmt is missing. (John Rouillard) +- save roundup-admin history between sessions. Load + ~/.roundup_admin_rlrc file to set history-size persistently. Add + pragma history_length to override for a session. (John Rouillard) +- the roundup-admin history command now dumps the journal entries + in a more human readable format. Use the raw option to get the older + machine parsible output. (John Rouillard) +- Multiple JWT secrets are supported to allow key rotation. See + an updated config.ini for details. (John Rouillard) +- issue2551212 - wsgi performance improvement feature added in 2.2.0 + is active by default. Can be turned off if needed. See upgrading.txt + for info. (John Rouillard) +- issue2551270 - Better templating support for JavaScript. Add + utils.readfile(file, optional=False) and utils.expandfile(file, + token_dict=None, optional=False). Allows reading an external file + (e.g. JavaScript) and inserting it using tal:contents or equivalent + jinja function. expandfile allows setting a dictionary and tokens in + the file of the form "%(token_name)s" will be replaced in the file + with the values from the dict. (John Rouillard) +- add @group to rest interface collection queries. Useful when using + optgroup in select elements. (John Rouillard) +- roundup-demo can set the hostname in the URL using the -H + parameter. So you can start a demo tracker that is available from + your network using 'roundup-demo ... -B hostname -H hostname'. (John + Rouillard) +- issue2551347 - make _generic.help.html work without property + settings. THis applies to classic or minimal trackers. It allows use + of classhelp without the property seting for informtion only + (e.g. description of what a priority or status means) without being + able to select the property in the classhelper. Good for adding help + for Link properties. (John Rouilllard) +- issue1525113 - notation to filter by logged-in user. Use + @current_user with properties that are a Link to the 'user' class to + match the currently logged in user. Allows sharing of queries like + "Issues I created" or "Issues I am assigned to" by removing the + hard coded user id number and replacing it with the current user's + id. Tracker templates updated to use it. (John Rouillard from a + patch by Jon C. Thomason) +- Add a /rest/data/user/roles REST endpoint. (John Rouillard) +- issue2551353 - Add roundup-classhelper for 2.4.0 + release. Integrate new classhelper web component to wrap + existing classhelper link. This fixes a number of + outstanding bugs against the current classhelper using + current web features. (Patel Malav, Nikunj Thakkar, + Bharath Kanama with integration by John Rouillard) +- disable spellcheck on all password fields to try to prevent + browser from exposing passwords to external servers. (John + Rouillard)
--- a/doc/installation.txt Tue Jul 09 22:13:25 2024 -0400 +++ b/doc/installation.txt Fri Jul 12 20:32:20 2024 -0400 @@ -304,11 +304,8 @@ .. _Using Redis for Session Databases: admin_guide.html#using-redis-for-session-databases -.. [3] Do not use Python 2 for new installs. The continuous - integration and other services used for developing Roundup - are dropping support for Python 2. Also optional packages - are dropping Python 2 support. As a result Python 2 may - not be supported for many more release cycles. +.. [3] Do not use Python 2 for new installs. The next minor release + (2.5.0 expected summer 2025) will drop support for Python 2. Installing Roundup ==================
--- a/doc/upgrading.txt Tue Jul 09 22:13:25 2024 -0400 +++ b/doc/upgrading.txt Fri Jul 12 20:32:20 2024 -0400 @@ -77,11 +77,9 @@ .. admonition:: Python 2 Support If you are running Roundup under Python 2, you should make plans to - switch to Python 3. Release 2.5.0 (expected release Jul 2025) - will be the last release to support Python 2. The continuous - Integration (CI) and other services - used for developing Roundup are dropping support for Python 2. Also - optional packages are dropping Python 2 support. + switch to Python 3. Release 2.4.0 (Jul 2024) is the last release to + officially support Python 2. The next non-patch release scheduled + for 2025 will mark 5 years since Roundup supported Python 3. .. admonition:: XHTML Support Deprecation Notice
--- a/website/www/index.txt Tue Jul 09 22:13:25 2024 -0400 +++ b/website/www/index.txt Fri Jul 12 20:32:20 2024 -0400 @@ -79,71 +79,86 @@ Roundup is highly customizable, allowing users to tailor the system to their specific needs and preferences. -The latest stable version of Roundup is 2.3.0, which includes bug -fixes and additional features compared to the previous 2.2.0 release. +The latest stable version of Roundup is 2.4.0, which includes bug +fixes and additional features compared to the previous 2.3.0 release. Roundup is compatible with Python 2.7.12+ or 3.6+. .. admonition:: Python 2 Support - Python 2 support will end with release 2.5.0 (expected July 2025). + Python 2 support ends with release 2.4.0 (July 2024). Several Continuous Integration (CI) services are - discontinuing their resources for testing with Python 2. It is - recommended to utilize Python 3 for the deployment of new trackers, - while existing trackers should be `upgraded to use Python 3. + discontinuing their resources for testing with Python 2. + Use Python 3 for the deployment of new trackers. + Existing trackers should be `upgraded to use Python 3. <docs/upgrading.html#python-3-support-info>`_ Release Highlights ================== -Some improvements from the 2.2.0 release are: - -* Dockerfile demo mode implemented. This allows quick evaluation as - well as the ability to spin up a configured tracker to customise. +Some improvements from the 2.3.0 release are: -* SQLite backends can use WAL mode to reduce blocking between readers - and writers improving concurrent use. - -* Redis can be used for session database with SQLite and dbm - backends. Provides a major performance improvement. +* three CVE's have been fixed. One requires changes to your + tracker's home directory. The other two are fixed by + installing 2.4.0. See + https://www.roundup-tracker.org/docs/security.html for + details and instructions on how to fix these in 2.4.0 and + earlier releases. -* roundup-mailgw can use OAUTH authentication to SMTP - server. (roundup-mailgw command line options changed as a result.) - -* Postgres full text index can now be enabled. +* new classhelper component thanks to a team of students + from CS682 at U-Mass Boston. This fixes many issues with + the old classhelper. It is implemented as a web-component + and needs REST interface access. It will fall back to the + classic classhelper if REST is not available or if the + browser does not support web-components. -* Modifications to in-reply-to threading when there are multiple - matches resulting in more predictable handling of messages. +* fix Windows Python installation using pip. It used to go + into an infinite loop during install or download. Also fix + installation of shared files (templates) so roundup-admin + can find them. -* Many updates to documentation to make it scannable, useful and - work on mobile. +* using ``@current_user`` as a value in a search URL for a + user property will use the current logged in user. Now you + can share searches like: "My issues" as "my" will become + the current logged in user. + +* login failures to the REST/XML-RPC interfaces are now rate + limited to limit password guessing attacks. -* Admin documentation includes a section on setting up Content - Security Policy (CSP) to better secure your Roundup trackers. - -* REST now allows rate limiting headers to be accessed by client - JavaScript. +* utf8mb4 is the default charset for MySQL. This requires + migrating your database using the mysql client. You can + choose to keep the older character set in config.ini. -* Default number of rounds for PBKDF2 updated to 2M to account for - improvements in password crackers and CPU power. +* PostgreSQL services defined in pg_service.conf can be + used. PostgreSQL schemas are supported to eliminate the + need for the roundup user to have database + creation/deletion privileges. -* Support PBKDF2 with SHA512 for password storage to improve - resistance to password crackers. +* fix out of memory issue when importing larger trackers + into PostgreSQL. + +* multiple roundup-admin improvements: display protected + properties (like creation date), better formatting of + output, command history. Also on windows, pyreadline3 is + supported to provide an editable interactive command line. -* Deprecate SSHA password hash function. +* an experimental wsgi performance improvement in 2.3.0 is + now now the default and is opt-out. -* roundup-admin reindex can be done in batches to manage load - incurred by reindexing. +* new template functions: utils.readfile and + utils.expandfile. Javascript that is included in the + Python core will be moved to external files and be able to + have values from Roundup substituted in the Javascript. -* roundup-admin can list available templates and their installed - locations. This is useful when installing via pip or in a docker - container as supporting files are not stored in the usual locations - like /usr/share/roundup. +* allow content-type of a template to be set from inside the + template. This allows returning json or xml from a + template without a .json or .xml extention. -* Crash fixes in detector handling, configuration handling, fix for - sorting of multilinks. +* fix import/export on windows to use Unix style line + endings fixing export/import on Windows and making exports + portable across platforms. -More info on the 53 changes can be found in the `change notes`_. +More info on the 79 changes can be found in the `change notes`_. Roundup Use Cases ================= @@ -177,17 +192,17 @@ demo.py``" and load up the URL it prints out! Follow the source gratification mode with these steps (change the -``-2.3.0`` version identifier to match your downloaded file). +``-2.4.0`` version identifier to match your downloaded file). 1. ``python3 -m pip download roundup`` -2. ``tar -xzvf roundup-2.3.0.tar.gz`` +2. ``tar -xzvf roundup-2.4.0.tar.gz`` * if you don't have a tar command (e.g windows), use:: - python -c "import tarfile, sys; tarfile.open(sys.argv[1]).extractall();" roundup-2.3.0.tar.gz + python -c "import tarfile, sys; tarfile.open(sys.argv[1]).extractall();" roundup-2.4.0.tar.gz -3. ``cd roundup-2.3.0`` +3. ``cd roundup-2.4.0`` 4. ``python3 demo.py`` (The source download can also be used to `create a custom Docker
