| changeset | 520075b29474 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: support justhtml parsing library to convert email to plain text justhtml is an pure python, fast, HTML5 compliant parser. It is now an option for converting html only emails to plain text. Its output format differs slightly from dehtml or beautifulsoup. Mostly by removing extra blank lines. dehtml.py: Using the stream parser of justhtml. Unable to get the full document parser to successfully strip script and style blocks. If I can fix this and use the standard parser, I can in theory generate markdown from the DOM tree generated by justhtml. Updated test case to include inline elements that should not cause a line break when they are encountered. Running dehtml as: `python roundup/dehtml.py foo.html` will load foo.html and parse it using all available parsers. configuration.py: justhtml is available as an option. docs: updated CHANGES.txt, doc/tracker_config.txt added beautifulsoup and justhtml to the optional software section of doc/installtion.txt. test_mailgw.py, .github/workflows/ci-test Updated tests and install justhtml as part of CI. |
| files |
| changeset | 7142740e6547 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge heads: CHANGES.txt also whitespace fixups. |
| files |
| changeset | b09ef85f0da6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: add nanoid pkg trace_id gen and decorator for setting processName nanoid is a shorter unique id generator and faster than uuid. I truncate nanoid id's to 12 chars to make it more readable. Also added decorator to allow setting the default processName definition in the logging module. admin.py and wsgi_handler now set processName. configuration.py knows how to overide the processName if set to the default MainProcess. Updated install docs to add nanoid as optional, how to switch to different trace_id output. pydoc generated docs include logcontext module and are referenced from admin.py. |
| files |
| changeset | f7fadbac5856 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | take2: merge multiple fix: python < 3.12 returns ParsingError not RuntimeError; print exception |
| files |
| changeset | 9f7e863fd1cd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge multiple fix: python < 3.12 returns ParsingError not RuntimeError; print exception |
| files |
| changeset | 6d6f689d3701 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: python < 3.12 returns ParsingError not RuntimeError; print exception RuntimeError added in 3.12. Support older configparser.ParsingError. Supporting older version also requires different test string for the two versions. Also use str(exception) rather than exception.args[0] for producing message to user. Include exception name in output. |
| files |
| changeset | 588ae856cd57 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: python 3.{7,8,10} raise configfile.ParsingError not RuntimeError when adding broken syntax: =foo to logging config file. These version failing in CI. 3.9 and 3.11 are also affected. RuntimeError was added in 3.12. |
| files |
| changeset | e91ff70e4563 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | bug: improve error reporting for errors for logging fileConfig. |
| files |
| changeset | 401c6f0be6c5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | bug: fix json logging config file syntax exception/fix test for windows If the json logging config file has mimatched {} or [], it raises an IndexError. Handle that case and test it. Also handle embedded filenames in tests when testsare run on windows:(/ vs \ directory sep). |
| files |
| changeset | d06be9346c68 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | bug, test: fix tests for trace_id; readd import logging.config Made save_restore_logging a test level fixture. It was a class level which worked fine until I started using caplog for tests in the same class. Due to loading config from dict, the roundup channel was set to not propagate which broke the new formatting test used for trace_id. Forgot to update some tests due to change in default format adding %(trace_id). Also re-added logging.config import which broke loading logging config files in configuration.py. |
| files |
| changeset | 14c7c07b32d8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feature: add thread local trace_id and trace_reason to logging. Added trace_id to default logging so that all logs for a given request share the same trace_id. This allows correlation of logs across a request. admin_guide.txt, upgrading.txt: add docs update sample configs to include trace_id. rewrite logging docs in admin_guide. Hopefully they are clearer now. clean up some stuff in the logging config file docs. admin.py: add decorators to run_command to enable trace_id. change calls to db.commit() to use run_command to get trace_id. configuration.py: clean up imports. update docstrings, comments and inline docs. add trace_id to default log format. add function for testing decorated with trace_id. add support for dumping stack trace in logging. add check for pytest in sys.modules to enable log propagation when pytest is running. Otherwise tests fail as the caplog logger doesn't see the roundup logs. logcontext.py: new file to handle thread local contextvar mangement. mailgw.py: add decorators for trace_id etc. scripts/roundup_xlmrpc_server.py: add decorators for trace_id etc. fix encoding bug turning bytes into a string. fix command line issue where we can't set encoding. (not sure if changing encoding via command line even works) cgi/client.py decorate two entry points for trace_id etc. cgi/wsgi_handler.py: decorate entry point for trace_id etc. test/test_config.py: add test for trace_id in new log format. test various cases for sinfo and errors in formating msg. |
| files |
| changeset | 39a6825d10ca |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: allow admin to set logging format from config.ini This is prep work for adding a per thread logging variable that can be used to tie all logs for a single request together. This uses the same default logging format as before, just moves it to config.ini. Also because of configparser, the logging format has to have doubled % signs. So use: %%(asctime)s not '%(asctime)s' as configparser tries to interpolate that string and asctime is not defined in the configparser's scope. Using %%(asctime)s is not interpolated by configparser and is passed into Roundup. |
| files |
| changeset | 66284037142e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | refactor: also error on missing file or invalid extension Refactored the code to reuse check that logging config file is set and that the file exists. Now throws error and exits if file name does not end in .ini or .json. Now throws error if file doesn't exist. Before it would just configure default logging as though file wasn't specified. Added tests for these two cases. |
| files |
| changeset | de1dac9abcb6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: change comment in dictConfig json file to // from # Emacs json mode at least will properly indent when using // as a comment character and not #. |
| files |
| changeset | 94eed885e958 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: add support for using dictConfig to configure logging. Basic logging config (one level and one output file non-rotating) was always possible from config.ini. However the LOGGING_CONFIG setting could be used to load an ini fileConfig style file to set various channels (e.g. roundup.hyperdb) (also called qualname or tags) with their own logging level, destination (rotating file, socket, /dev/null) and log format. This is now a deprecated method in newer logging modules. The dictConfig format is preferred and allows disabiling other loggers as well as invoking new loggers in local code. This commit adds support for it reading the dict from a .json file. It also implements a comment convention so you can document the dictConfig. configuration.py: new code test_config.py: test added for the new code. admin_guide.txt, upgrading.txt CHANGES.txt: docs added upgrading references the section in admin_guid. |
| files |
| changeset | 7ff47307b4b1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551398 - Browser number input Disable use of native browser number inputs. There are issues including use of comma decimal separators. More extensive remditaion/granularity is needed for this so disable but leave available for people to test with for now. Also moved the implentation of native date type from fixed area to feature area because of the issues with the native date popup and it defaulting to off. This means it's not a bugfix for the various bugs lodged against the roundup date picker component. |
| files |
| changeset | b99e76e76496 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make native date and number elements configurable Now for Number() and Integer() properties the browser-native number format can be configured with the use_browser_number_input config item in seciont [web]. The default is 'yes'. For Date() properties the config item is use_browser_date_input (also in section [web]) but the default is 'no'. In addition when defining Date() properties, these now have a parameter 'display_time' which defaults to 'yes' and a 'format' parameter which defaults to None. These set defaults for the field() method of the DateHTMLProperty which have the same parameters (but the display_time parameter of field() takes a boolean, not 'yes'/'no'). |
| files |
| changeset | 6bd11a73f2ed |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551253. default hash is PBKDF2-SHA512. The default password hashing algorithm has been upgraded to PBKDF2-SHA512 from PBKDF2-SHA1. The default pbkdf2 rounds in the config file has been changed to 250000. Doc updated. |
| files |
| changeset | 5a2b9435a04d |
|---|---|
| branch | permission-performance |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make permission filter functions configurable For debugging and performance measurements it makes sense to allow turning permission filter functions off. |
| files |
| changeset | 69629a4f7262 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | doc: in description, replace field with header for csrf_* settings The doc for the csrf_* settings referered to "field is missing" when referencing it as headers elsewhere in the doc. Change to use header instead of field to reduce confusion. |
| files |
| changeset | 9ba60ddafd2a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: remove duplicate config option. Had the same config setting twice in the structure. |
| files |
| changeset | 791b61ed11c9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551323 - Remove XHTML support First step: raise error if xhtml used in config.ini. Back end code is still present to allow an admin to re-enable xhtml as a temporary measure. |
| files |
| changeset | 6aa0525187cd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | doc: use generated config.ini in reference.txt Deleted the manually maintained config.ini reference items. Lost a few nice index references along the way but... Updated configuration.py with longer descriptions from the manually maintained items where appropriate. Checking in the generated tracker_config.txt and the Makefile used to generate the config and the man pages and then generate local documentation using setup.py build_docs. |
| files |
| changeset | 82093eb944d6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | test - normalize path for MultiFilePathOptions As part of work for issue2551334 - get test suite running under windows use os.path.normpath on absolute path processing to provide similar processing to os.path.join on the relative codepath. |
| files |
| changeset | 8b31893f5930 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551115/issue2551282 - utf8mb4 support in roundup Fix issues with utf8 support in Roundup. By default using: utf8mb4 charset utf8mb4_unicode_ci collation (case insensitive) utf8mb4_0900_ci collation (case sensitive) which are settable from config.ini. Sadly I couldn't come up with a way to mange these from one parameter. Doing a compatibility lookup table would have increased the maintenance burden and have me chasing MySQL changes. So I opted for the easy path and have the admins (with more MySQL experience) make the choices. Conversion directions added to upgrading.txt. I don't have any good testing for this. I was able to generate utf8/utf8mb3 tables and load a little data and convert. However this is a poor substitute for a conversion on a working tracker 8-(. |
| files |
| changeset | 0c71ac9cdcd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | chore(lint): change class props from mutable lists to tuples A number of class props were lists [] which are mutable. Change them to tuples. Better performance and no change of getting the contents changed. |
| files |
| changeset | b95474b23440 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | chore(lint): whitespace fixes, double space between class stanzas |
| files |
| changeset | ecc34b7917e2 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | chore(refactor): multiple changes/cleanups Simplify 'for x in [list val]; alist.append(x)' to 'alist = list([list val])'. Easier to read as copy of list. Also twice as fast although speed not an issue. Remove unneeded list() wrappers. Replace set(list comprehension) with set comprehension. Also add trailing ,'s to last element in tuples/lists. Add some noqa items for acceptable operations in context. Switch " ... \" ...\" ..." to: ' ... " ... " ...' to remove need to escape internal '"'. Change 'not x in y' to 'x not in y'. |
| files |
| changeset | a4b71f16c264 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | chore(refactor): use 'with open'; if ternarys; unnested if statements |
| files |
| changeset | 890097bc4cd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | chore(ruff): sort imports |
| files |
| changeset | be6cb2e0d471 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: add support for rotating jwt keys This allows jwt_secret to have multiple ',' separated secrets. The first/leftmost should be used to sign new JWTs. All of them are used (starting from left/newest) to try to verify a JWT. If the first secret is < 32 chars in length JWTs are disabled. If any of the other secrets are < 32 chars, the configuration code causes the software to exit. This prevents insecure (too short) secrets from being used. Updated doc examples and tests. |
| files |
| changeset | 6f66d74d37f3 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add configurable logging for REST We now log status code and error message for failing REST requests. Introduces new config item rest_logging in section [web]. Fixes (part of) issue2551274. |
| files |
| changeset | 3071db43bfb6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: issue2550852 - support using a specified PostgreSQL db schema Finally after 7 years this is closed. roundup/backends/back_postgresql.py: Support use of schema when specified in RDBMS_NAME. Stuart McGraws code is finally merged 8-). test/test_postgresql.py, test/conftest.py: Run all postgresql tests in the schema db as well. Also make sure that db_nuke raises an error when trying to delete the schema test database. Conftest defines pg_schema mark that can be used to exclude schema tests with pytest -m "not pg_schema". roundup/configuration.py: change doc on RDBMS_NAME to include db.schema form. .travis.yml, .github/workflows/ci-test.yml: create schema test db; add user for testing with schema; grant new user create privs for schema. doc/installation.txt: Reference to roundup-admin init deleting schema added. doc/mysql.txt doc/postgresql.txt: New documentation on psql/mysql commands to set up a production db. doc/upgrading.txt: mention schema support, also document service setting for selecting connection from pg_service.conf. doc/reference.txt: update config.ini documentation for RDBMS_NAME. |
| files |
| changeset | 4af0d235b570 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat(db): support using postgresql service connection file Add new service rdbms config option to set the service name to be used with a postgresql service connection file. This can be done using the PGSERVICE environment variable for a single instance tracker server. For a multi-instance server this per-tracker config option is needed. Note that settings (host, user, (db)name...) in config.ini file will override the service connection file setting. Also setting PGSERVICE and service will use the service setting. |
| files |
| changeset | d30e534b078a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | clarify doc on dispatcher_email config setting. An issue was brought up on the mailing list. https://sourceforge.net/p/roundup/mailman/message/43383465/ The description of dispatcher_email sounds like it should be sent email on issue creation. That's not it's role. Try to make it's role more obvious. Fix config.ini and reference.txt description. Add the newissuecopy.py detector to send email on the creation of an issue |
| files |
| changeset | 5e118944ef75 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | flake8: add extra blank lines |
| files |
| changeset | 273c8c2b5042 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix(api): - issue2551063 - Rest/Xmlrpc interfaces needs failed login protection. Failed API login rate limiting with expiring lockout added. |
| files |
| changeset | bed28b64c581 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add xhtml deprecation notice. xhtml is dead. Plan on removing it in next release along with the code that supports it. |
| files |
| changeset | a210f4437b49 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Incomplete work to generate config doc from config.ini This is an incomplete attempt to allow generation of the config.ini documentation in reference.txt. It reformats the output of 'roundup_admin.py genconfig'. So it now includes all of the settings. Using a Makefile rule like: tracker_config.txt: ../roundup/configuration.py python3 ../roundup/scripts/roundup_admin.py \ genconfig _temp_config.txt sed -e '1,8d' \ -e 's/^\[\([a-z]*\)\]/\n.. index:: config.ini; sections \1\n\n.. code:: ini\n\n [\1]/' \ -e 's/^\([^[]\)/ \1/' \ _temp_config.txt > tracker_config.txt rm -f _temp_config.txt results in the config.ini split by section and index links being put in place. However some sections have a comment before the [section] marker. This comment is orphaned at the end of the prior section rather than starting the new section. A simple sed won't allow the lookahead needed to target the [section] marker and include the prior comment block. Also there are still have some long lines generated (> 65 characters). Maybe a python script can import configuration.py and output proper restructured text output? reference.txt: add a commented out include:: tracker_config.txt directive roundup/admin.py: don't require a tracker home for genconfig. So user can generate a clean config.ini on demand. Tracker home is still required for updateconfig. roundup/configuration.py: wrap lines better. A number of them are generating comments > 65 characters which is the targeted line length. This cleans up config.ini too, so is a good thing. website/www/conf.py: ignore doc/tracker_config.ini when processing. |
| files |
| changeset | ab0ea9f03866 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix initialization of option_validator When using option_validator to validate config.ini entries for detectors or extensions, the list of validators was not reset when the tracker was re-opened. This led to the second command run in roundup-admin crashing when the main config.ini file was read. On second tracker open/read option_validator lists an extension/detector option that doesn't exist in the main config.ini. When it tried to validate, the missing option caused a crash. This only happens if a validator is added in an extension/detector. If the only validators ae ones for the main config.ini everything works fine across multiple tracker opens. |
| files |
| changeset | 5c71b27aa68e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix typo |
| files |
| changeset | 2de72f75f2f8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Production PBKDF rounds back to 2M, test 1k; fix empty_form (python2) |
| files |
| changeset | e3f3f859256c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | revert increase in PBKDF2 rounds to get baseline perf. |
| files |
| changeset | be7849588372 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551252 - increase PBKFD2 default rounds to 2,000,000. Current https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2 for SHA1 recommends 1,300,000 so 2,000,000. |
| files |
| changeset | 89a59e46b3af |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | improve REST interface security When using REST, we reflect the client's origin. If the wildcard '*' is used in allowed_api_origins all origins are allowed. When this is done, it also added an 'Access-Control-Allow-Credentials: true' header. This Credentials header should not be added if the site is matched only by '*'. This header should be provided only for explicit origins (e.g. https://example.org) not for the wildcard. This is now fixed for CORS preflight OPTIONS request as well as normal GET, PUT, DELETE, POST, PATCH and OPTIONS requests. A missing Access-Control-Allow-Credentials will prevent the tracker from being accessed using credentials. This prevents an unauthorized third party web site from using a user's credentials to access information in the tracker that is not publicly available. Added test for this specific case. In addition, allowed_api_origins can include explicit origins in addition to '*'. '*' must be first in the list. Also adapted numerous tests to work with these changes. Doc updates. |
| files |
| changeset | 563f5327c5b5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | clarify help text for * in allowed_api-origins |
| files |
| changeset | 570abc4c6548 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Improve documention on access to templates and static_files. |
| files |
| changeset | 8733aa2a8e40 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | flake8 fixes. |
| files |
| changeset | 9ff091537f43 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | postgresql native-fts; more indexer tests 1) Make postgresql native-fts actually work. 2) Add simple stopword filtering to sqlite native-fts indexer. 3) Add more tests for indexer_common get_indexer Details: 1) roundup/backends/indexer_postgresql_fts.py: ignore ValueError raised if we try to index a string with a null character in it. This could happen due to an incorrect text/ mime type on a file that has nulls in it. Replace ValueError raised by postgresql with customized IndexerQueryError if a search string has a null in it. roundup/backends/rdbms_common.py: Make postgresql native-fts work. When specified it was using using whatever was returned from get_indexer(). However loading the native-fts indexer backend failed because there was no connection to the postgresql database when this call was made. Simple solution, move the call after the open_connection call in Database::__init__(). However the open_connection call creates the schema for the database if it is not there. The schema builds tables for indexer=native type indexing. As part of the build it looks at the indexer to see the min/max size of the indexed tokens. No indexer define, we get a crash. So it's a a chicken/egg issue. I solved it by setting the indexer to the Indexer from indexer_common which has the min/max token size info. I also added a no-op save_indexer to this Indexer class. I claim save_indexer() isn't needed as a commit() on the db does all the saving required. Then after open_connection is called, I call get_indexer to retrieve the correct indexer and indexer_postgresql_fts woks since the conn connection property is defined. roundup/backends/indexer_common.py: add save_index() method for indexer. It does nothing but is needed in rdbms backends during schema initialization. 2) roundup/backends/indexer_sqlite_fts.py: when this indexer is used, the indexer test in DBTest on the word "the" fail. This is due to missing stopword filtering. Implement basic stopword filtering for bare stopwords (like 'the') to make the test pass. Note: this indexer is not currently automatically run by the CI suite, it was found during manual testing. However there is a FIXME to extract the indexer tests from DBTest and run it using this backend. roundup/configuration.py, roundup/doc/admin_guide.txt: update doc on stopword use for sqlite native-fts. test/db_test_base.py: DBTest::testStringBinary creates a file with nulls in it. It was breaking postgresql with native-fts indexer. Changed test to assign mime type application/octet-stream that prevents it from being processed by any text search indexer. add test to exclude indexer searching in specific props. This code path was untested before. test/test_indexer.py: add test to call find with no words. Untested code path. add test to index and find a string with a null \x00 byte. it was tested inadvertently by testStringBinary but this makes it explicit and moves it to indexer testing. (one version each for: generic, postgresql and mysql) Renamed Get_IndexerAutoSelectTest to Get_IndexerTest and renamed autoselect tests to include autoselect. Added tests for an invalid indexer and using native-fts with anydbm (unsupported combo) to make sure the code does something useful if the validation in configuration.py is broken. test/test_liveserver.py: add test to load an issue add test using text search (fts) to find the issue add tests to find issue using postgresql native-fts test/test_postgresql.py, test/test_sqlite.py: added explanation on how to setup integration test using native-fts. added code to clean up test environment if native-fts test is run. |
| files |
| changeset | 3f60a71b0812 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Support selecion session/otk data store. Add redis as data store. Allow admin to select the backend data store. Compatibility matrix: main\/ session>| anydbm | sqlite | redis | mysql | postgresql | anydbm | D | | X | | | sqlite | X | D | X | | | mysql | | | | D | | postgresql | | | | | D | --------------------------------------------------------------+ D - default if unconfigured, X - compatible choice DETAILS roundup/configuration.py: add config.ini section sessiondb with settings: backend and redis_url. CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt: doc on config of session db and redis. Plus some other fixes: admin - clarified why we do not drop __words and __testids table in native-fts conversion. TYpo fix. upgrading - doc how you can keep using anydbm for session data with sqlite. Fix dupe sentence in an upgrading config.ini section. roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py: code to support redis, redis/anydbm backends respectively. roundup/backends/sessions_redis.py new storage backend for redis. roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py redis uses a different way of calculating lifetime/timestamp. Since expiration of an item occurred if its timestamp was more than 1 week old, code would calculate: now - 1 week + lifetime. But this results in faster expiration in redis if used for lifetime/timestamp. Convert code to use the lifetime() method in BasicDatabase that generates the right timestamp for each backend. test/session_common.py: added tests for more cases, get without default, getall non-existing key etc. timestamp test changed to use new self.get_ts which is overridden in other tests. Test that datatypes survive storage. test/test_redis_session.py: test redis session store with sqlite and anydbm primary databases test/test_anydbm.py, test/test_sqlite.py add test to make sure the databases are properly set up sqlite - add test cases where anydbm is used as datastore anydbm - remove updateTimestamp override add get_ts(). test/test_config.py tests on redis_url and compatibility on choice of sessiondb backend .travis.yml: add redis db and redis-py |
| files |
| changeset | 031996eb9bb5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | document jinja2 as valid setting for template_engine in config.ini Also try to import jinja2 and report unavailable/available. |
| files |
| changeset | f1f2d59dab8b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add allowed_api_origins to upgrading doc In upgrading doc, also make upgrading config.ini a separate step. Fix spelling error. Also document * in allowed_api_origins in configuration.py. |
| files |
| changeset | 9ca5cbffa0c4 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Switch off using blank passwords for login There is now a config.ini setting [web] login_empty_passwords to enable logins for users without a password set. By default it's off and every user must have a password. |
| files |
| changeset | 0b8d34b64930 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551205 bugfix missing _value2str |
| files |
| changeset | ab2ed11c021e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551205: Add support for specifying valid origins for api: xmlrpc/rest We now have an allow list to filter the hosts allowed to do api requests. An element of this allow list must match the http ORIGIN header exactly or the rest/xmlrpc CORS request will result in an error. The tracker host is always allowed to do a request. |
| files |
| changeset | 408fd477761f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add i18n object to roundupdb.Database This makes the i18n object accessible everywhere (including in detectors where localized error messages were impossible). See issue2551184 |
| files |
| changeset | 0d99ae7c8de6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Allow Roundup to use PostgreSQL database native full text search back_postgreql.py - schema version changes for schema version 7. configuration.py - added indexer_language checks for postgresql. Hardcoded list for now. Docs admin_guide and upgrading Tests. This also restructures the version upgrade tests for the rdbms backends. They can run all of them now as the proper cascade is developed to roll back changes to version 6. |
| files |
| changeset | 91ab3e0ffcd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Add test cases for sqlite fts Add support for using the FTS5 full text query engine for sqlite. Also stubbed out some sections for adding postgresql FTS support as well. Added nee indexer type native-fts. It is not selected by default. The indexer=native is used if no indexer is set. This prevents an upgrade from seeming to wipe out the native index if upgraded and indexer=native is not explicitly set. Docs updated. Also changed section headers to sentence case for the current release notes. Indexing backend can control if the full text search phrase is broken into a list of words or passed intact. For backends with query languages (sqlite and can be enabled for whoosh and xapian) we do not want the phrase "tokenized" on whitespace. This also updates the rdbms database version to version 7 to add FTS table. I will be using the same version when I add postgresql. If somebody runs this version on postgresql, they will have to manually add the fts tables for postgresql if they want to use it. Added a new renderError method to client. This allows errors to be reported still using page.html rather than raw html. It also supports templates for any error code. If no template for the error code (e.g. 400) is found, the error in raw html with no page frame is shown. New IndexerQueryError exception to pass back message about query syntax errors. |
| files |
| changeset | 770503bd211e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Validate SecretOption and support validate method Needed to validate SecretOption and verify that file is readable and valid. validator() now calls the validate method for each Option subclass. To add post config load validation, just define a method validate(self, options) for the Option subclass. |
| files |
| changeset | b1f1539c6a31 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551182 - ... allow loading values from external file. flake8 cleanups Secrets (passwords, secrets) can specify a file using file:// or file:///. The first line of the file is used as the secret. This allows committing config.ini to a VCS. Following settings are changed: [tracker] secret_key [tracker] jwt_secret [rdbms] password [mail] password details: in roundup/configuration.py: Defined SecretMandatoryOptions and SecretNullableOptions. Converted all secret keys and password to one of the above. Also if [mail] username is defined but [mail] password is not it throws an error at load. Cleaned up a couple of methods whose call signature included: def ...(..., settings={}): settings=None and it is set to empty dict inside the method. Also replace exception.message with str(exception) for python3 compatibility. in test/test_config: changed munge_configini to support changing only within a section, replacing keyword text. |
| files |