| changeset | 94aafe3b0a0d |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551334 - get more tests working under windows Fix indexer tests for file based db's running under windows. Apparently if there is an open/uncommitted cursor sqlite3db.close() doesn't actually close the file under windows. Because these tests run isolated indexer functions, the commits aren't done in the code under test. |
| files |
| changeset | 5be92f16a684 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | test: fix test_indexer for change from list to tuples in config. |
| files |
| changeset | d17e57220a62 |
|---|---|
| branch | |
| bookmark | |
| tag | 2.3.1a0 |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: close file properly in indexer_dbm.py:save_index() Fix this error found in debug logs of gentoo packaging of round 2.2.0. /roundup/backends/indexer_dbm.py:253: ResourceWarning: unclosed file <_io.BufferedWriter name='test-index/indexes/index.db-'> open(self.indexdb+'-', 'wb').write(zlib.compress(marshal.dumps(dbfil))) Also added test that calls save_index(), reloads the index and tests that the original item. I am not sure how Gentoo hit this But they were missing a number of backends. So it's possible that indexer_dbm.py is not getting fully tested depending on what is installed on the system. Codecov from CI didnt show indexer_dbm.py:save_index() being covered. |
| files |
| changeset | 67e7225c4343 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | test: fix sqlite indexer test Had an import psycopg2 that was not neded or used. It shouldn't have been there. Probbly bad copy/paste. Also remove @skip_postgres that isn't needed now. |
| files |
| changeset | 890b8f4ff38e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add a missing @skip_postgresql |
| files |
| changeset | c6b2534a58a9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Compatibility: unittest.mock vs. only mock Python3 has unittest.mock while python2 has only mock. At least on the systems tested (with installed unittest). |
| files |
| changeset | 9ff091537f43 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | postgresql native-fts; more indexer tests 1) Make postgresql native-fts actually work. 2) Add simple stopword filtering to sqlite native-fts indexer. 3) Add more tests for indexer_common get_indexer Details: 1) roundup/backends/indexer_postgresql_fts.py: ignore ValueError raised if we try to index a string with a null character in it. This could happen due to an incorrect text/ mime type on a file that has nulls in it. Replace ValueError raised by postgresql with customized IndexerQueryError if a search string has a null in it. roundup/backends/rdbms_common.py: Make postgresql native-fts work. When specified it was using using whatever was returned from get_indexer(). However loading the native-fts indexer backend failed because there was no connection to the postgresql database when this call was made. Simple solution, move the call after the open_connection call in Database::__init__(). However the open_connection call creates the schema for the database if it is not there. The schema builds tables for indexer=native type indexing. As part of the build it looks at the indexer to see the min/max size of the indexed tokens. No indexer define, we get a crash. So it's a a chicken/egg issue. I solved it by setting the indexer to the Indexer from indexer_common which has the min/max token size info. I also added a no-op save_indexer to this Indexer class. I claim save_indexer() isn't needed as a commit() on the db does all the saving required. Then after open_connection is called, I call get_indexer to retrieve the correct indexer and indexer_postgresql_fts woks since the conn connection property is defined. roundup/backends/indexer_common.py: add save_index() method for indexer. It does nothing but is needed in rdbms backends during schema initialization. 2) roundup/backends/indexer_sqlite_fts.py: when this indexer is used, the indexer test in DBTest on the word "the" fail. This is due to missing stopword filtering. Implement basic stopword filtering for bare stopwords (like 'the') to make the test pass. Note: this indexer is not currently automatically run by the CI suite, it was found during manual testing. However there is a FIXME to extract the indexer tests from DBTest and run it using this backend. roundup/configuration.py, roundup/doc/admin_guide.txt: update doc on stopword use for sqlite native-fts. test/db_test_base.py: DBTest::testStringBinary creates a file with nulls in it. It was breaking postgresql with native-fts indexer. Changed test to assign mime type application/octet-stream that prevents it from being processed by any text search indexer. add test to exclude indexer searching in specific props. This code path was untested before. test/test_indexer.py: add test to call find with no words. Untested code path. add test to index and find a string with a null \x00 byte. it was tested inadvertently by testStringBinary but this makes it explicit and moves it to indexer testing. (one version each for: generic, postgresql and mysql) Renamed Get_IndexerAutoSelectTest to Get_IndexerTest and renamed autoselect tests to include autoselect. Added tests for an invalid indexer and using native-fts with anydbm (unsupported combo) to make sure the code does something useful if the validation in configuration.py is broken. test/test_liveserver.py: add test to load an issue add test using text search (fts) to find the issue add tests to find issue using postgresql native-fts test/test_postgresql.py, test/test_sqlite.py: added explanation on how to setup integration test using native-fts. added code to clean up test environment if native-fts test is run. |
| files |
| changeset | a23eaa3013e6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | test get_indexer three autosearch options: xapian, whoosh, native |
| files |
| changeset | 86cde9cae7e1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Improve test coverage of backends/indexer_common::get_indexer() method |
| files |
| changeset | 3260926d7e7e |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix postgresl-fts indexer: get_indexer reported not implemented When I implemented postgresql fts indexing I forgot to remove code that raised NotImplemented when it was requested. Also had to fix import to match filename. Added tests for native-fts paths in get_indexer. |
| files |
| changeset | 0d99ae7c8de6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Allow Roundup to use PostgreSQL database native full text search back_postgreql.py - schema version changes for schema version 7. configuration.py - added indexer_language checks for postgresql. Hardcoded list for now. Docs admin_guide and upgrading Tests. This also restructures the version upgrade tests for the rdbms backends. They can run all of them now as the proper cascade is developed to roll back changes to version 6. |
| files |
| changeset | 39189dd94f2c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551189 - increase size of words in full text index. Increased indexed word maxlength to 50 DB migration code is written and tests work. Restructured some tests to allow for code reuse. Docs. If this passes CI without errors 2551189 should be done. However, testing on my system generates errors. Encoding (indexer unicode russian unicode string invalid) and collation errors (utf8_bin not valid) when running under python2. No issues with python3 and I haven't changed code that should cause these since the last successful build in CI. So if this fails in CI we will have more checkins. |
| files |
| changeset | 91ab3e0ffcd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Add test cases for sqlite fts Add support for using the FTS5 full text query engine for sqlite. Also stubbed out some sections for adding postgresql FTS support as well. Added nee indexer type native-fts. It is not selected by default. The indexer=native is used if no indexer is set. This prevents an upgrade from seeming to wipe out the native index if upgraded and indexer=native is not explicitly set. Docs updated. Also changed section headers to sentence case for the current release notes. Indexing backend can control if the full text search phrase is broken into a list of words or passed intact. For backends with query languages (sqlite and can be enabled for whoosh and xapian) we do not want the phrase "tokenized" on whitespace. This also updates the rdbms database version to version 7 to add FTS table. I will be using the same version when I add postgresql. If somebody runs this version on postgresql, they will have to manually add the fts tables for postgresql if they want to use it. Added a new renderError method to client. This allows errors to be reported still using page.html rather than raw html. It also supports templates for any error code. If no template for the error code (e.g. 400) is found, the error in raw html with no page frame is shown. New IndexerQueryError exception to pass back message about query syntax errors. |
| files |
| changeset | c26b9ce33ae3 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551123 - validate indexer_language in configuration.py Was validated in backends/indexer_xapian.py which would throw an error on access rather than on start. Added validator function to CoreConfig class that runs after config.ini is read. At this time we have access to the indexer setting so can determine if xapian is actually going to be used. Moved test into test/test_config.py and pulled validation code from indexer_xapian.py and test/test_indexer.py. |
| files |
| changeset | 9d209d2b34ae |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add indexer_language to change stemmer for xapian FTS indexer Nagy Gabor asked how to enable the hungarian stemmer in roundup. This required editing indexer_xapian.py replacing hardcoded "english" term. This value is now exposed in the config file under [main] index_language. This only works for xapian currently. |
| files |
| changeset | 5bf7b5debb09 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Fix xapian indexer for unicode Make regular expression pattern match words as unicode. |
| files |
| changeset | 6137ea845438 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Fix russian search/string russian string is supposed to be two words. Use only 1 of the words with find(). |
| files |
| changeset | f6c58a7b535c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add a russian string for unicode testing of indexer Also assign strings to vars to make it easier to mix/match tests in the future. |
| files |
| changeset | 0db2621b6fee |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add test for issue1344046 and maybe issue1195739 Python3 should be properly indexing unicode words while python2 doesn't. Add test (xfail for python 2) for this. So far I have passing on python3 (tested 4 of 6 indexers) and fail on 2 of 6 and xpass on 2 of 6 under python2. |
| files |
| changeset | f8893e1cde0d |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | assert_ is depricated. Replacing with assertTrue to reduce logs in travisci. |
| files |
| changeset | d26921b851c3 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Joseph Myers <jsm@polyomino.org.uk> |
| description | Python 3 preparation: make relative imports explicit. Tool-generated patch. |
| files |
| changeset | 43a1f7fe39f5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Improved work-around for pytest markers bug The previous fix was only a partial solution. Any test class sharing a parent with, and appearing after, a skipped test class was being skipped (not just other test classes using the skip/skipif marker). Now only tests that should be skipped will be skipped, the rest should run as normal. |
| files |
| changeset | 37d1e24fb941 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Fix work-around for pytest markers bug The initial work-around implemented was totally botched using 'pytest.skip' instead of 'pytest.mark.skip' which resulted in all tests in a file being completely ignored if any skip conditions that evaluated to true were declared or imported in the file. This work-around will not correctly display why all the tests have been skipped when using the '-rs' parameter. Only the first skip marker to taint a parent test class will be displayed (ie. if both xapian and mysql tests are being skipped, pytest will only output that tests are being skipped because xapian is not installed even though the mysql tests are also being skipped because mysql backend is not available). There also seems to be a bug in the current version of pytest being used in 'run_tests.py' (v2.8.4) that results in the skip not actually working when using 'pytest.mark.skip'. This does work correctly with the most recent release (v2.9.2), so the 'run_tests.py' script will need to be updated. |
| files |
| changeset | e74c3611b138 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | - issue2550636, issue2550909: Added support for Whoosh indexer. Also adds new config.ini setting called indexer to select indexer. See ``doc/upgrading.txt`` for details. Initial patch done by David Wolever. Patch modified (see ticket or below for changes), docs updated and committed. I have an outstanding issue with test/test_indexer.py. I have to comment out all imports and tests for indexers I don't have (i.e. mysql, postgres) otherwise no tests run. With that change made, dbm, sqlite (rdbms), xapian and whoosh indexes are all passing the indexer tests. Changes summary: 1) support native back ends dbm and rdbms. (original patch only fell through to dbm) 2) Developed whoosh stopfilter to not index stopwords or words outside the the maxlength and minlength limits defined in index_common.py. Required to pass the extremewords test_indexer test. Also I removed a call to .lower on the input text as the tokenizer I chose automatically does the lowercase. 3) Added support for max/min length to find. This was needed to pass extremewords test. 4) Added back a call to save_index in add_text. This allowed all but two tests to pass. 5) Fixed a call to: results = searcher.search(query.Term("identifier", identifier)) which had an extra parameter that is an error under current whoosh. 6) Set limit=None in search call for find() otherwise it only return 10 items. This allowed it to pass manyresults test Also due to changes in the roundup code removed the call in indexer_whoosh to from roundup.anypy.sets_ import set since we use the python builtin set. |
| files |
| changeset | c977f3530944 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Work-around for pytest.mark.skipif() bug There is a pytest bug that can cause all test classes marked with the skipif() decorator that inherit a common class to be skipped if one of the skipif() conditions is True. See: https://github.com/pytest-dev/pytest/issues/568 |
| files |
| changeset | 364c54991861 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Remove unneeded TestSuite code from tests The TestSuite code is no longer needed now that we are using py.test which can automatically discover tests |
| files |
| changeset | 380d8d8b30a3 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Replace existing run_tests.py script with a pytest script The existing run_test.py script is quite old, a bit restrictive, and doesn't always behave as documented. The pytest testing tool is mature, well documented, and maintained. The run_tests.py script is generating by installing py.test and running: py.tests --genscript=run_tests.py Note: to generate a script that is compatible with python2.6 the command needs to be run using python2.6 |
| files |
| changeset | 63c79c0992ae |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Kristensen <john@jerrykan.com> |
| description | Update tests to work with py.test py.test searches for any class that looks like a TestCase in the test directory and tries to run them as tests. Some of the classes that inherit TestCase are not meant to be run and are only intended to be "helper classes". Only the tests of the classes that inherit the "helper classes" should be run. If we convert these "helper classes" to be "mixins" py.test should not pick them up. |
| files |
| changeset | 3ff1a288fb9c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Thomas Arendsen Hein <thomas@intevation.de> |
| description | issue2550583, issue2550635 Do not limit results with Xapian indexer Other indexer backends do not limit the number of results. Add test with searching for 123 entries on all backends, more should not be used, because the slower backends would take too much time. enquire.get_mset was limited to 10 results and since the results can be messages instead of issues, the 10 results could even be just one issue if there were many messages matching the search term before searching the messages of other issues. Additionally the few results could be filtered by other attributes, e.g. only showing open issues, which caused even less matches. |
| files |
| changeset | 6e3e4f24c753 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Eric S. Raymond <esr@thyrsus.com> |
| description | Remove keyword expansions from CVS. All regression tests passed afterwards. |
| files |
| changeset | b41a033bffcc |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | - add a small word-splitting test for the indexers... when answering an mailinglist-mail concerning indexer behaviour |
| files |
| changeset | 2b1241daaa20 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Bernhard Reiter <Bernhard.Reiter@intevation.de> |
| description | Added more indexer tests for stopwords, case-insensitity... ...and short and long words. |
| files |
| changeset | dcca66d56815 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix unit test compatibility |
| files |
| changeset | 96a5e0845adb |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | now I understand the comment in that method... ...after my change it failed for python2.5. Now it works for all of them. |
| files |
| changeset | 6eec11b197aa |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
| description | fix for indexer-test: Old version fails for me on Debian Etch with sqlite 2.8.17-2. |
| files |
| changeset | 0bf9f8ae7d1b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix bug introduced in 1.4.5 in RDBMS full-text indexing; fix tests so the code is actually exercised; fix another bug discovered in process |
| files |
| changeset | 7728ee93efd2 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | fix reindexing in Xapian |
| files |
| changeset | a4edd24c32be |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | test fixes and checking of indexer overwrites (xapian currently fails) |
| files |
| changeset | 8f7dc283bfa5 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | some more Xapian stuff (doc, test fixes) |
| files |
| changeset | a615cc230160 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | added Xapian indexer; replaces standard indexers if Xapian is available |
| files |
| changeset | 2fee36128471 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Johannes Gijsbers <jlgijsbers@users.sourceforge.net> |
| description | Fix test to account for changed interface to find(). |
| files |
| changeset | 9c8de04a76b1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | unit test fixes |
| files |
| changeset | 0e1860dcb328 |
|---|---|
| branch | maint-0.7 |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | unit test fixes |
| files |
| changeset | 93f03c6714d8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | A few big changes in this commit: 1. The current indexer has been moved to backends/indexer_dbm in anticipation of my writing an indexer_rdbms, 2. Changed indexer invocation during create / set to follow the pattern set by the metakit backend, which was much cleaner, and 3. The "content" property of FileClass is now mutable in all but the metakit backend. Metakit needs to be changed to support the editing of "content". Hey, and I learnt today that the metakit backend implements its own indexer. How about that... :) |
| files |
| changeset | f63aa57386b0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | Backend improvements. - using Zope3's test runner now, allowing GC checks, nicer controls and coverage analysis - all RDMBS backends now have indexes on several columns - added testing of schema mutation, fixed rdbms backends handling of a couple of cases - !BETA! added postgresql backend, needs work !BETA! |
| files |
| changeset | 9b910e8d987d |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | removed Log |
| files |
| changeset | 2a928d404af8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Richard Jones <richard@users.sourceforge.net> |
| description | ehem, forgot to add |
| files |