| changeset | 95f91b6f0386 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551343 - Remove support for PySQLite and sqlite v2. It is unmaintained and sqlite3 is used which is the default for a Python distribution. |
| files |
| changeset | e3975f679bf1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551302 - Remove support for sqlite version 1 from back_sqlite.py Remove sqlite v1 support and document. |
| files |
| changeset | e10d0b262abd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: diagnose/report use of SQLite without FTS5 support As of 2.2.0 Roundup requires FTS5 support in SQLite. FTS5 has been part of the main SQLite distribution since October 2015. Tonu Mikk found this when trying to run 2.3.0 under RedHat 7. He got a traceback when trying to spin up 2.3.0. It took us a bit to figure out that FTS5 was missing from the SQLite library used by Python3. See: https://sourceforge.net/p/roundup/mailman/message/51783129/ This change catches the sql error and checks to see if the ENABLE_FTS5 compile option is defined. If not it raises NotImplementedError with a more useful error message and reports the version of SQLite in use. This will at least ease diagnosis. Trying to support SQLite without FTS5 support raises a number of issues including tracking the internal schema used by Roundup. So not going to attempt that. Details: https://sourceforge.net/p/roundup/mailman/message/51783321/ |
| files |
| changeset | e138f5cf432a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix comment on use of sqlite for session db's. |
| files |
| changeset | ff7f6c0bbdae |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | flake8 fixes. |
| files |
| changeset | a96a239db0d9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Set all sqlite db's to WAL mode on creation Complete work done on 6917:fba76e0bba98. WAL journaling mode now enabled on primary db as well as session db's when using sqlite for session databases. Made change to WAL mode more robust for testing by bracketing `pragma journal_mode=wal` with commit() calls. Normally the conversion would occur when a new session was opened, but this commits the changes explicitly to make all the tests stable. Also added doc on this. |
| files |
| changeset | fba76e0bba98 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | set journal mode to wal when creating db. Trying to work around sqlite3.OperationalError: disk I/O error errors in CI that I can't reproduce locally. Also this allows reads to happen while writes are occurring. This should permit using the web interface for browsing while a large write load (e.g. bulk data loading from another tracker) is happening. Right now autogenerating issues locks up the tracker for reading with an error that the database is locked. |
| files |
| changeset | fad6b3a272cd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add instrumentation to debug I/O error in CI that I don't get locally try to debug 'sqlite3.OperationalError: disk I/O error' failures: https://github.com/roundup-tracker/roundup/actions/runs/2995869962 https://app.travis-ci.com/github/roundup-tracker/roundup/jobs/581894108 during commit(). Get info on mode, uid, gid etc. for db directory and files in it. I can provoke the failure by stopping before a self.conn.commit(), chmod 555 the directory holding the db then continue. But even running a full test suite I never see this error. It only occurs for some tests and not others and doesn't seem to be consistent. One other possible workaround is setting it to WAL mode rather than journal mode when opening it. |
| files |
| changeset | feb175271f3e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | redis import failure on python2 causes crash Don't use ModuleNotFoundError, it's python 3.6+. Use plain ImportError instead. |
| files |
| changeset | 3f60a71b0812 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Support selecion session/otk data store. Add redis as data store. Allow admin to select the backend data store. Compatibility matrix: main\/ session>| anydbm | sqlite | redis | mysql | postgresql | anydbm | D | | X | | | sqlite | X | D | X | | | mysql | | | | D | | postgresql | | | | | D | --------------------------------------------------------------+ D - default if unconfigured, X - compatible choice DETAILS roundup/configuration.py: add config.ini section sessiondb with settings: backend and redis_url. CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt: doc on config of session db and redis. Plus some other fixes: admin - clarified why we do not drop __words and __testids table in native-fts conversion. TYpo fix. upgrading - doc how you can keep using anydbm for session data with sqlite. Fix dupe sentence in an upgrading config.ini section. roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py: code to support redis, redis/anydbm backends respectively. roundup/backends/sessions_redis.py new storage backend for redis. roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py redis uses a different way of calculating lifetime/timestamp. Since expiration of an item occurred if its timestamp was more than 1 week old, code would calculate: now - 1 week + lifetime. But this results in faster expiration in redis if used for lifetime/timestamp. Convert code to use the lifetime() method in BasicDatabase that generates the right timestamp for each backend. test/session_common.py: added tests for more cases, get without default, getall non-existing key etc. timestamp test changed to use new self.get_ts which is overridden in other tests. Test that datatypes survive storage. test/test_redis_session.py: test redis session store with sqlite and anydbm primary databases test/test_anydbm.py, test/test_sqlite.py add test to make sure the databases are properly set up sqlite - add test cases where anydbm is used as datastore anydbm - remove updateTimestamp override add get_ts(). test/test_config.py tests on redis_url and compatibility on choice of sessiondb backend .travis.yml: add redis db and redis-py |
| files |
| changeset | 25d08e15e3b4 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551224 - Replace dbm db for sessions/otks when using sqlite Generate new sqlite db's for storing one time keys and session and other ephemeral data. |
| files |
| changeset | 39189dd94f2c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551189 - increase size of words in full text index. Increased indexed word maxlength to 50 DB migration code is written and tests work. Restructured some tests to allow for code reuse. Docs. If this passes CI without errors 2551189 should be done. However, testing on my system generates errors. Encoding (indexer unicode russian unicode string invalid) and collation errors (utf8_bin not valid) when running under python2. No issues with python3 and I haven't changed code that should cause these since the last successful build in CI. So if this fails in CI we will have more checkins. |
| files |
| changeset | 91ab3e0ffcd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Add test cases for sqlite fts Add support for using the FTS5 full text query engine for sqlite. Also stubbed out some sections for adding postgresql FTS support as well. Added nee indexer type native-fts. It is not selected by default. The indexer=native is used if no indexer is set. This prevents an upgrade from seeming to wipe out the native index if upgraded and indexer=native is not explicitly set. Docs updated. Also changed section headers to sentence case for the current release notes. Indexing backend can control if the full text search phrase is broken into a list of words or passed intact. For backends with query languages (sqlite and can be enabled for whoosh and xapian) we do not want the phrase "tokenized" on whitespace. This also updates the rdbms database version to version 7 to add FTS table. I will be using the same version when I add postgresql. If somebody runs this version on postgresql, they will have to manually add the fts tables for postgresql if they want to use it. Added a new renderError method to client. This allows errors to be reported still using page.html rather than raw html. It also supports templates for any error code. If no template for the error code (e.g. 400) is found, the error in raw html with no page frame is shown. New IndexerQueryError exception to pass back message about query syntax errors. |
| files |
| changeset | 19a58adf5d1f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Flake8: whitepace; don't import base64; unused loop var gets leading _ |
| files |
| changeset | 16e1255b16cf |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement limit and offset for filter These map to the corresponding SQL parameters for the SQL backends. |
| files |
| changeset | ee2e8f8d6648 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement exact string search .. in the 'filter' method of hyperdb.Class (and the corresponding backend implementations). |
| files |
| changeset | 56c9bcdea47f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Joseph Myers <jsm@polyomino.org.uk> |
| description | Python 3 preparation: unicode. This patch introduces roundup/anypy/strings.py, which has a comment explaining the string representations generally used and common functions to handle the required conversions. Places in the code that explicitly reference the "unicode" type / built-in function are generally changed to use the new functions (or, in a few places where those new functions don't seem to fit well, other approaches such as references to type(u'') or use of the codecs module). This patch does not generally attempt to address text conversions in any places not currently referencing the "unicode" type (although scripts/import_sf.py is made to use binary I/O in places as fixing the "unicode" reference didn't seem coherent otherwise). |
| files |
| changeset | 62de601bdf6f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix commits although a Reject exception is raised Fix the problem that changes are committed to the database (due to commits to otk handling) even when a Reject exception occurs. The fix implements separate database connections for otk/session handling and normal database operation. |
| files |
| changeset | 506c7ee9a385 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add a 'retired' parameter to Class.filter .. to allow searching for retired, non-retired or all (retired and non-retired) items similar to the argument of the same name to Class.getnodeids. This is 'False' by default (finding only non-retired items for backwards compatibility) and can be set to None (for finding retired and non-retired items) or True (for finding only retired items). |
| files |
| changeset | 198b6e810c67 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Eric S. Raymond <esr@thyrsus.com> |
| description | Use Python-3-compatible 'as' syntax for except statements Many raise statements near these are also fixed. So are two ivorrect file encoding marks ('utf8'->'utf-8'). |
| files |
| changeset | e1e40674a0bc |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement double-precision Number .. as an option to the 'Number' property. |
| files |
| changeset | 9cebf686c552 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2550727: db.newid is broken with sqlite. Added proper transaction lock around the sql code to get a new id. The the locking that pysqlite attempts had to be defeated because it is broken. Had to explicitly manage transactions with BEGIN IMMEDIATE and call sql_commit. Note that this reduces performance by 30% in return for accuracy. Also use update call set newid=newid+1 rather than incrementing in python. |
| files |
| changeset | e74c3611b138 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | - issue2550636, issue2550909: Added support for Whoosh indexer. Also adds new config.ini setting called indexer to select indexer. See ``doc/upgrading.txt`` for details. Initial patch done by David Wolever. Patch modified (see ticket or below for changes), docs updated and committed. I have an outstanding issue with test/test_indexer.py. I have to comment out all imports and tests for indexers I don't have (i.e. mysql, postgres) otherwise no tests run. With that change made, dbm, sqlite (rdbms), xapian and whoosh indexes are all passing the indexer tests. Changes summary: 1) support native back ends dbm and rdbms. (original patch only fell through to dbm) 2) Developed whoosh stopfilter to not index stopwords or words outside the the maxlength and minlength limits defined in index_common.py. Required to pass the extremewords test_indexer test. Also I removed a call to .lower on the input text as the tokenizer I chose automatically does the lowercase. 3) Added support for max/min length to find. This was needed to pass extremewords test. 4) Added back a call to save_index in add_text. This allowed all but two tests to pass. 5) Fixed a call to: results = searcher.search(query.Term("identifier", identifier)) which had an extra parameter that is an error under current whoosh. 6) Set limit=None in search call for find() otherwise it only return 10 items. This allowed it to pass manyresults test Also due to changes in the roundup code removed the call in indexer_whoosh to from roundup.anypy.sets_ import set since we use the python builtin set. |
| files |
| changeset | e424987d294a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add support for an integer type to join the existing number type. Commit patch supplied for issue2550886. This can be used for properties used for ordering, counts etc. where a decimal point isn't needed. Developed by Anthony (antmail). Doc updates written by John Rouillard. |
| files |
| changeset | 3adff0fb0207 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Thomas Arendsen Hein <thomas@intevation.de> |
| description | Fixed issue2550595: Allow migrating from roundup 0.x to 1.4 All changes were required to make an upgrade from 0.6 to 1.4, The changes affecting "retired" were required for an upgrade from 0.8 to 1.4. |
| files |
| changeset | 67bef70ab9b9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | - more logger fixes, sorry for the noise. |
| files |
| changeset | 9655a1b65974 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | - more logger fixes -- use correct hierarchical logger names... ...always starting with "roundup." -- otherwise logger configuration changed in changeset 2010-08-09T03:36:06Z!schlatterbeck@users.sourceforge.net will not work (e.g. turning debugging on) |
| files |
| changeset | 3e35233ea93c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | new rdbms config item sqlite_timeout... ...makes the previously hard-coded timeout of 30 seconds configurable. This is the time a client waits for the locked database to become free before giving up. Used only for SQLite backend. |
| files |
| changeset | 13b3155869e0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Beginnings of a big code cleanup / modernisation to make 2to3 happy |
| files |
| changeset | 289f249ba192 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | compatibilty fix for pysqlite2 / unicode |
| files |
| changeset | 23ba17d01a24 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | re-order sqlite imports to handle multiple installed versions (issue 2550570) |
| files |
| changeset | 09e79cbeb827 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | force sqlite3 in py2.6+ to treat our stored text as UTF-8 |
| files |
| changeset | 59d02edfaa9c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Stefan Seefeld <stefan@seefeld.name> |
| description | Uniformly use """...""" instead of '''...''' for comments. |
| files |
| changeset | bb30bbfc7cdd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | Indexing fixes. - Add new rdbms-indexes to the tables for the full-text index. These speed up adding new entries and finding old ones to remove. - Remove getprops method from FileClass in backends/rdbms_common: This forced indexing for the content property even if the user decided to turn it off for a FileClass instance. |
| files |
| changeset | f5ddd1639a15 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | *** empty log message *** |
| files |
| changeset | c92687dce135 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | unicode / sqlite 3 problem [SF#1589292] |
| files |
| changeset | b476fef16ccc |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fixed support for pysqlite2 (version 2.1.0 is the minimum version supported) |
| files |
| changeset | 0d561b24ceff |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | support sqlite3 |
| files |
| changeset | a775afeeee8b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | Database interface fixes. - Fix a problem with wrong sort-order in RDBMS backend when a sort-attribute is also searched for. We need to preserve correct sort attribute order in that case. A test for this was added earlier. - Added forgotten fail_ok parameter to sql_commit method in sqlite backend to make tests work again. - Note that this version still fails a test for the postgresql backend. |
| files |
| changeset | f5201bbbc505 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | update for latest version of pysqlite ([SF#1487098]; patch 1534227) |
| files |
| changeset | f2fda3e6fc8b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | umask is now configurable (with the same 0002 default) |
| files |
| changeset | 9020fe58c51c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Anthony Baxter <anthonybaxter@users.sourceforge.net> |
| description | more errors. backport candidate |
| files |
| changeset | 3518d1ffd940 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from maint-0-7 |
| files |
| changeset | 088c1792b7f6 |
|---|---|
| branch | maint-0.8 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from maint-0-7 |
| files |
| changeset | c7c25f2103b2 |
|---|---|
| branch | maint-0.7 |
| bookmark | |
| tag | 0.7.12 |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix RDBMS clear() so it resets all class itemid counters |
| files |
| changeset | ae1093684ef5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from maint-0-7 |
| files |
| changeset | e074c641cb5e |
|---|---|
| branch | maint-0.7 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix RDBMS clear() so it resets all class itemid counters |
| files |
| changeset | a9b0fe809d16 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | remove unnecessary import |
| files |
| changeset | 43675a5232b1 |
|---|---|
| branch | maint-0.8 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | remove unnecessary import |
| files |
| changeset | e1da7b5b04ab |
|---|---|
| branch | maint-0.8 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from HEAD |
| files |
| changeset | 57b60bda9473 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Python 2.3 minimum version - bye bye roundup.rlog, you had a short life. |
| files |
| changeset | d9b4224f955c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from maint-0-8 |
| files |
| changeset | d3a5e382d44a |
|---|---|
| branch | maint-0.7 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from maint-0-8 |
| files |
| changeset | a7045bad20de |
|---|---|
| branch | maint-0.8 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | have RDBMS full-text indexer do AND searching [SF#1055435] |
| files |
| changeset | 240a24f1e69f |
|---|---|
| branch | maint-0.8 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from HEAD |
| files |
| changeset | c5530df3aaa6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | back_sqlite.py is missing "import time" [SF#1081959] |
| files |
| changeset | fddbbeb581bc |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Alexander Smishlajev <a1s@users.sourceforge.net> |
| description | sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x). have to install sqlite_busy_handler instead. |
| files |
| changeset | d3b3f1b3d59e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Alexander Smishlajev <a1s@users.sourceforge.net> |
| description | sqlite database creation requires existing directory. create the database directory in sql_open_connection() if the directory does not exist. |
| files |
| changeset | 402d6d556558 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | postgres backend open doesn't hide corruption in schema [SF#956375] *dbm-style backends nuke() method now clear id counters |
| files |
| changeset | 8140fb128088 |
|---|---|
| branch | maint-0.7 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | merge from HEAD |
| files |