| changeset | 39c482e6a246 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: fix code to make tests of session and otks databases pass on windows The existing code had a few issues: sessions_dbm.py: detect dumbdbm when used on windows python and properly delete session/otks databases so clear() works. make sure the Session/Otks.cache_db_type() is called even when the database is newly created. test/session_common.py: close session and otks database in teardown before deleting database directory to prevent errors from deleting open files on windows. test/test_sqlite.py: close the session and otks databases opened by SessionTest.setUp(self) so the salite session/otks database are closed before the new anydbm sessions databases are opened for testing the sqlite main db and anydbm as session/otks db. Again this causes deletion of database test directory to fail on windows as you can't delete open files. |
| files |
| changeset | 521d98231e5c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Test self.db.db_version_updated in schema upgrade paths A user on the roundup-users mailing list reported that a 'roundup-admin migrate' reported that changes were needed. But it should have done the migration. roundup-admin uses the property db.db_version_updated to tell if there was a migration. Testing never looked at that property. Add a test for that property in all migration updates. AFAICT the test is always successful. So no bug uncovered by these tests. |
| files |
| changeset | a96a239db0d9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Set all sqlite db's to WAL mode on creation Complete work done on 6917:fba76e0bba98. WAL journaling mode now enabled on primary db as well as session db's when using sqlite for session databases. Made change to WAL mode more robust for testing by bracketing `pragma journal_mode=wal` with commit() calls. Normally the conversion would occur when a new session was opened, but this commits the changes explicitly to make all the tests stable. Also added doc on this. |
| files |
| changeset | 22d001cafd09 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | try calling tearDown and see if tests run correctly. I had DBTest.tearDown not DBTest.tearDown(). Sigh. See if I get failures in CI with this change. |
| files |
| changeset | 3cfb8eccc86e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Remove sqlite setup/teardown methods does this fix the sqlite OperationalError failures? |
| files |
| changeset | 9ff091537f43 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | postgresql native-fts; more indexer tests 1) Make postgresql native-fts actually work. 2) Add simple stopword filtering to sqlite native-fts indexer. 3) Add more tests for indexer_common get_indexer Details: 1) roundup/backends/indexer_postgresql_fts.py: ignore ValueError raised if we try to index a string with a null character in it. This could happen due to an incorrect text/ mime type on a file that has nulls in it. Replace ValueError raised by postgresql with customized IndexerQueryError if a search string has a null in it. roundup/backends/rdbms_common.py: Make postgresql native-fts work. When specified it was using using whatever was returned from get_indexer(). However loading the native-fts indexer backend failed because there was no connection to the postgresql database when this call was made. Simple solution, move the call after the open_connection call in Database::__init__(). However the open_connection call creates the schema for the database if it is not there. The schema builds tables for indexer=native type indexing. As part of the build it looks at the indexer to see the min/max size of the indexed tokens. No indexer define, we get a crash. So it's a a chicken/egg issue. I solved it by setting the indexer to the Indexer from indexer_common which has the min/max token size info. I also added a no-op save_indexer to this Indexer class. I claim save_indexer() isn't needed as a commit() on the db does all the saving required. Then after open_connection is called, I call get_indexer to retrieve the correct indexer and indexer_postgresql_fts woks since the conn connection property is defined. roundup/backends/indexer_common.py: add save_index() method for indexer. It does nothing but is needed in rdbms backends during schema initialization. 2) roundup/backends/indexer_sqlite_fts.py: when this indexer is used, the indexer test in DBTest on the word "the" fail. This is due to missing stopword filtering. Implement basic stopword filtering for bare stopwords (like 'the') to make the test pass. Note: this indexer is not currently automatically run by the CI suite, it was found during manual testing. However there is a FIXME to extract the indexer tests from DBTest and run it using this backend. roundup/configuration.py, roundup/doc/admin_guide.txt: update doc on stopword use for sqlite native-fts. test/db_test_base.py: DBTest::testStringBinary creates a file with nulls in it. It was breaking postgresql with native-fts indexer. Changed test to assign mime type application/octet-stream that prevents it from being processed by any text search indexer. add test to exclude indexer searching in specific props. This code path was untested before. test/test_indexer.py: add test to call find with no words. Untested code path. add test to index and find a string with a null \x00 byte. it was tested inadvertently by testStringBinary but this makes it explicit and moves it to indexer testing. (one version each for: generic, postgresql and mysql) Renamed Get_IndexerAutoSelectTest to Get_IndexerTest and renamed autoselect tests to include autoselect. Added tests for an invalid indexer and using native-fts with anydbm (unsupported combo) to make sure the code does something useful if the validation in configuration.py is broken. test/test_liveserver.py: add test to load an issue add test using text search (fts) to find the issue add tests to find issue using postgresql native-fts test/test_postgresql.py, test/test_sqlite.py: added explanation on how to setup integration test using native-fts. added code to clean up test environment if native-fts test is run. |
| files |
| changeset | 75a5946cf897 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix tests that were leaving test dir behind. Zero out session/otks variables. don't try to set them to valid values. |
| files |
| changeset | 3f60a71b0812 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Support selecion session/otk data store. Add redis as data store. Allow admin to select the backend data store. Compatibility matrix: main\/ session>| anydbm | sqlite | redis | mysql | postgresql | anydbm | D | | X | | | sqlite | X | D | X | | | mysql | | | | D | | postgresql | | | | | D | --------------------------------------------------------------+ D - default if unconfigured, X - compatible choice DETAILS roundup/configuration.py: add config.ini section sessiondb with settings: backend and redis_url. CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt: doc on config of session db and redis. Plus some other fixes: admin - clarified why we do not drop __words and __testids table in native-fts conversion. TYpo fix. upgrading - doc how you can keep using anydbm for session data with sqlite. Fix dupe sentence in an upgrading config.ini section. roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py: code to support redis, redis/anydbm backends respectively. roundup/backends/sessions_redis.py new storage backend for redis. roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py redis uses a different way of calculating lifetime/timestamp. Since expiration of an item occurred if its timestamp was more than 1 week old, code would calculate: now - 1 week + lifetime. But this results in faster expiration in redis if used for lifetime/timestamp. Convert code to use the lifetime() method in BasicDatabase that generates the right timestamp for each backend. test/session_common.py: added tests for more cases, get without default, getall non-existing key etc. timestamp test changed to use new self.get_ts which is overridden in other tests. Test that datatypes survive storage. test/test_redis_session.py: test redis session store with sqlite and anydbm primary databases test/test_anydbm.py, test/test_sqlite.py add test to make sure the databases are properly set up sqlite - add test cases where anydbm is used as datastore anydbm - remove updateTimestamp override add get_ts(). test/test_config.py tests on redis_url and compatibility on choice of sessiondb backend .travis.yml: add redis db and redis-py |
| files |
| changeset | bdd28b244839 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | - issue2551223 - fix timestamp truncation in mysql and postgresql The data types used to represent timestamps in pg and mysql for ephemeral tables: sessions and otks don't have enough signifcant digits to work. As a result the timestamps are rounduped (up/down) rsuling in the stored timestamp being 2 minutes (pg) or 2-3 hours(mysql) off from what it should be. Modify db schema to use a numeric type that preserves more significant figures. Implement schema upgrade. Document need for upgrade in upgrading.txt. Write tests for schema upgrade. Implement test for updateTimestamp method on BasicDatabase that showed this issue in the first place. Write overrides for test for anydbm/memorydb which store timestamp properly or not at all. |
| files |
| changeset | 044dcf3608a2 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | update session db tests session_common.py: testList had no asserts. While adding them I found out the memory and anydbm backends return byte strings while the rdbms backend return strings. So added a call to s2b defined in each db test file to covert. rdbms i a no-op and memory/anydbm call roundup.anypy.strings::s2b(). also add some data to other tests and verify it. other files: define s2b appropriately. |
| files |
| changeset | 0d99ae7c8de6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Allow Roundup to use PostgreSQL database native full text search back_postgreql.py - schema version changes for schema version 7. configuration.py - added indexer_language checks for postgresql. Hardcoded list for now. Docs admin_guide and upgrading Tests. This also restructures the version upgrade tests for the rdbms backends. They can run all of them now as the proper cascade is developed to roll back changes to version 6. |
| files |
| changeset | 39189dd94f2c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551189 - increase size of words in full text index. Increased indexed word maxlength to 50 DB migration code is written and tests work. Restructured some tests to allow for code reuse. Docs. If this passes CI without errors 2551189 should be done. However, testing on my system generates errors. Encoding (indexer unicode russian unicode string invalid) and collation errors (utf8_bin not valid) when running under python2. No issues with python3 and I haven't changed code that should cause these since the last successful build in CI. So if this fails in CI we will have more checkins. |
| files |
| changeset | fcbeff272828 |
|---|---|
| branch | REST-rebased |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Integrate REST tests into db framework |
| files |
| changeset | d26921b851c3 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Joseph Myers <jsm@polyomino.org.uk> |
| description | Python 3 preparation: make relative imports explicit. Tool-generated patch. |
| files |
| changeset | 62de601bdf6f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix commits although a Reject exception is raised Fix the problem that changes are committed to the database (due to commits to otk handling) even when a Reject exception occurs. The fix implements separate database connections for otk/session handling and normal database operation. |
| files |
| changeset | efb34cbdba7c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add (currently failing) test for atomic actions Actions via the web-interface can have several database-modifying instructions. These should be atomic, i.e., either all should go in or none. This is currently not the case due to commit calls from OTK handling (CSRF-protection). |
| files |
| changeset | 364c54991861 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Remove unneeded TestSuite code from tests The TestSuite code is no longer needed now that we are using py.test which can automatically discover tests |
| files |
| changeset | 63c79c0992ae |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Update tests to work with py.test py.test searches for any class that looks like a TestCase in the test directory and tries to run them as tests. Some of the classes that inherit TestCase are not meant to be run and are only intended to be "helper classes". Only the tests of the classes that inherit the "helper classes" should be run. If we convert these "helper classes" to be "mixins" py.test should not pick them up. |
| files |
| changeset | 6e3e4f24c753 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Eric S. Raymond <esr@thyrsus.com> |
| description | Remove keyword expansions from CVS. All regression tests passed afterwards. |
| files |
| changeset | 34dce76bb202 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | Multilink fixes and optimizations: - Optimisation: Late evaluation of Multilinks (only in rdbms backends): previously we materialized each multilink in a Node -- this creates an SQL query for each multilink (e.g. 'files' and 'messages' for each line in the issue index display) -- even if the multilinks aren't displayed. Now we compute multilinks only if they're accessed (and keep them cached). - Add a filter_iter similar to the existing filter call. This feature is considered experimental. This is currently not used in the web-interface but passes all tests for the filter call except sorting by Multilinks (which isn't supported by SQL and isn't a sane concept anyway). When using filter_iter instead of filter this saves a *lot* of SQL queries: Filter returns only the IDs of Nodes in the database, the additional content of a Node has to be fetched in a separate SQL call. The new filter_iter also returns the IDs of Nodes (one by one, it's an iterator) but pre-seeds the cache with the content of the Node. The information needed for seeding the cache is retrieved in the same SQL query as the ids. |
| files |
| changeset | 2784c239e6c8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | clear the cache on commit for rdbms backends: Don't carry over cached values from one transaction to the next (there may be other changes from other transactions) see new ConcurrentDBTest for a read-modify-update cycle that fails with the old caching behavior. |
| files |
| changeset | 0bf9f8ae7d1b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix bug introduced in 1.4.5 in RDBMS full-text indexing; fix tests so the code is actually exercised; fix another bug discovered in process |
| files |
| changeset | adec352e2ce0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | don't try to import all backends in backends.__init__ unless we *want* to |
| files |
| changeset | c091cacdc505 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Finished implementation of session and one-time-key stores for RDBMS backends. Refactored the API of sessions and their interaction with the backend database a fair bit too. Added some session tests. Nothing testing ageing yet, 'cos that's a pain inna ass to test :) Note: metakit backend still uses the *dbm implementation. It might want to implement its own session store some day, as it'll be faster than the *dbm one. |
| files |
| changeset | f9316d2cd5ba |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Fixed retirement of items in rdbms imports [SF#841355] Fixed bug in looking up journal of newly-created items in *dbm backends |
| files |
| changeset | 043e1b699047 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | more unit test work |
| files |
| changeset | f63aa57386b0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Backend improvements. - using Zope3's test runner now, allowing GC checks, nicer controls and coverage analysis - all RDMBS backends now have indexes on several columns - added testing of schema mutation, fixed rdbms backends handling of a couple of cases - !BETA! added postgresql backend, needs work !BETA! |
| files |