Mercurial > p > roundup > code
view doc/acknowledgements.txt @ 6915:9ff091537f43
postgresql native-fts; more indexer tests
1) Make postgresql native-fts actually work.
2) Add simple stopword filtering to sqlite native-fts indexer.
3) Add more tests for indexer_common get_indexer
Details:
1) roundup/backends/indexer_postgresql_fts.py:
ignore ValueError raised if we try to index a string with a null
character in it. This could happen due to an incorrect text/ mime
type on a file that has nulls in it.
Replace ValueError raised by postgresql with customized
IndexerQueryError if a search string has a null in it.
roundup/backends/rdbms_common.py:
Make postgresql native-fts work. When specified it was using using
whatever was returned from get_indexer(). However loading the
native-fts indexer backend failed because there was no connection to
the postgresql database when this call was made.
Simple solution, move the call after the open_connection call in
Database::__init__().
However the open_connection call creates the schema for the
database if it is not there. The schema builds tables for
indexer=native type indexing. As part of the build it looks at the
indexer to see the min/max size of the indexed tokens. No indexer
define, we get a crash.
So it's a a chicken/egg issue. I solved it by setting the indexer
to the Indexer from indexer_common which has the min/max token size
info. I also added a no-op save_indexer to this Indexer class. I
claim save_indexer() isn't needed as a commit() on the db does all
the saving required. Then after open_connection is called, I call
get_indexer to retrieve the correct indexer and
indexer_postgresql_fts woks since the conn connection property is
defined.
roundup/backends/indexer_common.py:
add save_index() method for indexer. It does nothing but is needed
in rdbms backends during schema initialization.
2) roundup/backends/indexer_sqlite_fts.py:
when this indexer is used, the indexer test in DBTest on the word
"the" fail. This is due to missing stopword filtering. Implement
basic stopword filtering for bare stopwords (like 'the') to make the
test pass. Note: this indexer is not currently automatically run by
the CI suite, it was found during manual testing. However there is a
FIXME to extract the indexer tests from DBTest and run it using this
backend.
roundup/configuration.py, roundup/doc/admin_guide.txt:
update doc on stopword use for sqlite native-fts.
test/db_test_base.py:
DBTest::testStringBinary creates a file with nulls in it. It was
breaking postgresql with native-fts indexer. Changed test to assign
mime type application/octet-stream that prevents it from being
processed by any text search indexer.
add test to exclude indexer searching in specific props. This code
path was untested before.
test/test_indexer.py:
add test to call find with no words. Untested code path.
add test to index and find a string with a null \x00 byte. it was
tested inadvertently by testStringBinary but this makes it explicit
and moves it to indexer testing. (one version each for: generic,
postgresql and mysql)
Renamed Get_IndexerAutoSelectTest to Get_IndexerTest and renamed
autoselect tests to include autoselect. Added tests for an invalid
indexer and using native-fts with anydbm (unsupported combo) to make
sure the code does something useful if the validation in
configuration.py is broken.
test/test_liveserver.py:
add test to load an issue
add test using text search (fts) to find the issue
add tests to find issue using postgresql native-fts
test/test_postgresql.py, test/test_sqlite.py:
added explanation on how to setup integration test using native-fts.
added code to clean up test environment if native-fts test is run.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Mon, 05 Sep 2022 16:25:20 -0400 |
| parents | a1868fe784d0 |
| children | c83038a606ac |
line wrap: on
line source
Acknowledgements ================ The Roundup Initative is a group of people that regularly work together to produce new releases of the Roundup Issue Tracker. We like to thank our community and all organisation and people that support us doing so with code, money, time, testing, reports, ideas and everything else that helped! `Announcement with changelog for current release.`_ .. _`Announcement with changelog for current release.`: announcement.html 2.2 --- 2.2.0 ~~~~~ Maintainer: John Rouillard, Ralf Schlatterbeck Release Manager: John Rouillard Developer activity by changesets:: rouilj@ieee.org 291 ************************************************ rsc@runtux.com 11 ** aburke 1 cmeerw@cmeerw.org 1 thomas@intevation.de 1 Other contributers Marcus Priesch, John Kristensen (jerrykan), Ashely Burke, Nagy Gabor, Norbert Schlemmer, Thomas Arendsen Hein 2.1 --- 2.1.0 ~~~~~ Maintainer: John Rouillard, Ralf Schlatterbeck Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 178 *************************************************** rsc at runtux.com 29 ******** cedric.krier at b2ck.com 6 ** cmeerw at cmeerw.org 1 john at jerrykan.com 1 Tobias Herp 1 2.0 --- 2.0.0 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 62 *************************************************** rsc at runtux.com 11 ********* cmeerw at cmeerw.org 4 *** 2.0.0b0 ~~~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 133 ************************************************ cmeerw at cmeerw.org 35 ************* rsc at runtux.com 10 **** jsm at polyomino.org.uk 1 2.0.0a0 ~~~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj@ieee.org 13674 *************************************************************************************** jsm@polyomino.org.uk 4746 ****************************** cmeerw@cmeerw.org 3622 *********************** dangchau1991@yahoo.com 3374 ********************* rsc@runtux.com 2871 ****************** ezio.melotti@gmail.com 103 * john@jerrykan.com 40 bernhard@intevation.de 29 martin.v.loewis 8 cedric.krier@b2ck.com 6 tekberg@uw.edu 6 gabriele.roeger@unibas.ch 4 Other contributers Chau Nguyen, Andreas (anrounham14), Robert Klonner, Garth Jensen 1.6 --- 1.6.1 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 19 ********************************************* bernhard at intevation.de 1 ** jsm at polyomino.org.uk 1 ** Joseph Myers, Cédric Krier, hannob/JulienPalard 1.6.0 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 234 ********************************************** rsc at runtux.com 30 ****** john at jerrykan.com 26 ***** bernhard at intevation.de 10 ** esr at thyrsus.com 8 ** techtonik at gmail.com 5 * dangchau1991 at gmail.com 1 pf at artcom-gmbh.de 1 thomas at intevation.de 1 Frank Niessink, Anthony (antmail), Bruce Tulloch, kinggreedy, David Wolever, Tom Ekberg, Peter Funk, Daniel Diniz, W. Trevor King, Igor Ippolitov, Joseph Myers, r.david.murray, martin.v.loewis, Henry (henryl), David Wolever, Stuart McGraw (smcgraw), Philipp Gortan (mephinet), Ezio Melotti, Heiko Stegmann, Trent Gamblin, Anton Schur 1.5 --- 1.5.1 ~~~~~ Maintainer: Ralf Schlatterbeck Release Manager: Anatoly Techtonik Developer activity by changesets:: hg churn -c --rev 1.5.0:1.5.1 techtonik@gmail.com 98 ********************************************** bernhard@intevation.de 44 ********************* Bernhard Reiter rsc@runtux.com 37 ***************** Ralf Schlatterbeck john@jerrykan.com 26 ************ John Kristensen thomas@intevation.de 22 ********** Thomas Arendsen Hein rouilj@ieee.org 5 ** John P. Rouillard caulagi@gmail.com 2 * Pradip Caulagi ced@b2ck.com 1 Cédric Krier Mathias Behrle, Ezio Melotti, R David Murray, James Mack, Thibault Fevry, Kai Storbeck, Kay Hayen, Karl-Philipp Richter 1.5.0 ~~~~~ Maintainer: Ralf Schlatterbeck:: hg churn -c --rev 1.4.21:1.5.0 techtonik@gmail.com 37 ********************************************** Anatoly Techtonik bernhard@intevation.de 17 ********************* Bernhard Reiter caulagi@gmail.com 16 ******************** Pradip Caulagi rouilj@ieee.org 8 ********** John P. Rouillard rsc@runtux.com 6 ******* Ralf Schlatterbeck thomas@intevation.de 5 ****** Thomas Arendsen Hein Tom Ekberg, Erik Hanspers pre 1.5.0 --------- Go Ping, you rock! Also, go Common Ground, ekit.com and Bizar Software for letting me implement this system on their time. Thanks also to the many people on the mailing list, in the sourceforge project and those who just report bugs: Christian Aastorp, Thomas Arendsen Hein, Nerijus Baliunas, Benni Bärmann, Anthony Baxter, David Benjamin, Marlon van den Berg, Bo Berglund, Olly Betts, Stéphane Bidoul, Rafal Bisingier, Cameron Blackwood, Jeff Blaine, Duncan Booth, Seb Brezel, J Alan Brogan, Titus Brown, Steve Byan, Brett Cannon, Godefroid Chapelle, Pradip Caulagi, Eli Collins, Roch'e Compaan, Wil Cooley, Joe Cooper, Kelley Dagley, Bruno Damour, Bradley Dean, Toby Dickenson, Paul F. Dubois, Hauke Duden, Eric Earnst, Peter Eisentraut, Andrew Eland, Jeff Epler, Tom Epperly, Tamer Fahmy, Thibault Févry, Vickenty Fesunov, Hernan Martinez Foffani, Trent Gamblin, Stuart D. Gathman, Martin Geisler, Ajit George, Dirk Geschke, Frank Gibbons, Johannes Gijsbers, Christian Glass, Gus Gollings, Philipp Gortan, Dan Grassi, Robin Green, Jason Grout, Charles Groves, Engelbert Gruber, Bruce Guenter, Tamás Gulácsi, Satchidanand Haridas, Sebastian Harl, Ralf Hemmecke, Juergen Hermann, Tobias Herp, Uwe Hoffmann, Alex Holkner, Tobias Hunger, Werner Hunger, Simon Hyde, Paul Jimenez, Christophe Kalt, Timo Kankare, Brian Kelley, James Kew, Sheila King, Michael Klatt, Bastian Kleineidam, Robert Klonner, Axel Kollmorgen, Cédric Krier, John Kristensen, Detlef Lannert, Andrey Lebedev, Henrik Levkowetz, David Linke, Martin v. Löwis, Fredrik Lundh, Jochen Maes, Will Maier, Ksenia Marasanova, Georges Martin, Gordon McMillan, Christof Meerwald, John F Meinel Jr, Roland Meister, Ezio Melotti, Ulrik Mikaelsson, John Mitchell, Ramiro Morales, Toni Mueller, Joseph Myers, Chau Nguyen, Stefan Niederhauser, Truls E. Næss, Bryce L Nordgren, Patrick Ohly, "om", Luke Opperman, Anthony Pankov, Eddie Parker, Will Partain, Timo Paulssen, Benjamin Pollack, Peter Pöml, Ewout Prangsma, Marcus Priesch, Eric S. Raymond, Bernhard Reiter, Roy Rapoport, John P. Rouillard, Luke Ross, Jesse Ruderman, Ollie Rutherfurd, Toby Sargeant, Giuseppe Scelsi, Ralf Schlatterbeck, Gregor Schmid, Florian Schulze, Klamer Schutte, Dougal Scott, Stefan Seefeld, Jouni K Seppänen, Jeffrey P Shell, Dan Shidlovsky, Joel Shprentz, Terrel Shumway, Emil Sit, Alexander Smishlajev, Nathaniel Smith, Leonardo Soto, Maciej Starzyk, Heiko Stegmann, Kai Storbeck, Mitchell Surface, anatoly techtonik, Jon C. Thomason, Mike Thompson, Hubert Touvet, Michael Twomey, Joseph E. Trent, Karl Ulbrich, Martin Uzak, Darryl VanDorp, J Vickroy, Timothy J. Warren, Jakub Wilk, William (Wilk), Tue Wennerberg, Matt Wilbert, Chris Withers, David Wolever, Cheer Xiao, Milan Zamazal.