Mercurial > p > roundup > code
annotate test/test_indexer.py @ 3634:57c66056ffe4
Implemented what I'll call for now "transitive searching"...
...using the filter method. The first idea was mentioned on the
roundup-users mailing list:
http://article.gmane.org/gmane.comp.bug-tracking.roundup.user/6909
We can now search for items which link transitively to other classes
using filter. An example is searching for all items where a certain user
has added a message in the last week:
db.issue.filter (None, {'messages.author' : '42', 'messages.date' : '.-1w;'})
or more readable (but not exactly semantically equivalent, if you're
searching for multiple users in this way it will fail, because string
searches are ANDed): {'messages.author.username':'ralf', ...
We can even extend this further, look for all items that were changed by
users belonging to a certain department (having the same supervisor -- a
property that is not in the user class in standard roundup) in the last
week, the filterspec would be:
{'messages.author.supervisor' : '42', 'messages.date' : '.-1w;'}
If anybody wants to suggest another name instead of transitive
searching, you're welcome.
I've implemented a generic method for this in hyperdb.py -- the backend
now implements _filter in this case. With the generic method, anydbm and
metakit should work (anydbm is tested, metakit breaks for other
reasons). A backend may chose to implement the real transitive filter
itself. This was done for rdbms_common.py. It now has an implementation
of filter that supports transitive searching by creating one big join in
the generated SQL query.
I've added several new regression tests to test for the new features.
All the tests (not just the new ones) run through on python2.3 and
python2.4 with postgres, mysql, sqlite, anydbm -- but metakit was
already broken when I started.
I've generated a tag before commit called 'rsc_before_transitive_search'
and will create the 'after' tag after this commit, so you can merge out
my changes if you don't like them -- if you like them I can remove the
tags.
.-- Ralf
| author | Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net> |
|---|---|
| date | Sat, 08 Jul 2006 18:28:18 +0000 |
| parents | 7728ee93efd2 |
| children | 0bf9f8ae7d1b |
| rev | line source |
|---|---|
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
1 # Copyright (c) 2002 ekit.com Inc (http://www.ekit-inc.com/) |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
2 # |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
3 # Permission is hereby granted, free of charge, to any person obtaining a copy |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
4 # of this software and associated documentation files (the "Software"), to deal |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
5 # in the Software without restriction, including without limitation the rights |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
6 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
7 # copies of the Software, and to permit persons to whom the Software is |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
8 # furnished to do so, subject to the following conditions: |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
9 # |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
10 # The above copyright notice and this permission notice shall be included in |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
11 # all copies or substantial portions of the Software. |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
12 # |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
13 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
14 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
15 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
16 # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
17 # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
18 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
19 # SOFTWARE. |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
20 |
|
3547
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
21 # $Id: test_indexer.py,v 1.10 2006-02-07 04:59:05 richard Exp $ |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
22 |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
23 import os, unittest, shutil |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
24 |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
25 class db: |
|
3546
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
26 class config(dict): |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
27 DATABASE = 'test-index' |
|
3546
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
28 config = config() |
|
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
29 config[('main', 'indexer_stopwords')] = [] |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
30 |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
31 class IndexerTest(unittest.TestCase): |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
32 def setUp(self): |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
33 if os.path.exists('test-index'): |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
34 shutil.rmtree('test-index') |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 os.mkdir('test-index') |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 os.mkdir('test-index/files') |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
37 from roundup.backends.indexer_dbm import Indexer |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
38 self.dex = Indexer(db) |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 self.dex.load_index() |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
40 |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
41 def test_basics(self): |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
42 self.dex.add_text(('test', '1', 'foo'), 'a the hello world') |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
43 self.dex.add_text(('test', '2', 'foo'), 'blah blah the world') |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
44 self.assertEqual(self.dex.find(['world']), [('test', '1', 'foo'), |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
45 ('test', '2', 'foo')]) |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
46 self.assertEqual(self.dex.find(['blah']), [('test', '2', 'foo')]) |
|
3078
2fee36128471
Fix test to account for changed interface to find().
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2977
diff
changeset
|
47 self.assertEqual(self.dex.find(['blah', 'hello']), []) |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
48 |
|
3547
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
49 def test_change(self): |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
50 self.dex.add_text(('test', '1', 'foo'), 'a the hello world') |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
51 self.dex.add_text(('test', '2', 'foo'), 'blah blah the world') |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
52 self.assertEqual(self.dex.find(['world']), [('test', '1', 'foo'), |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
53 ('test', '2', 'foo')]) |
|
3546
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
54 self.dex.add_text(('test', '1', 'foo'), 'a the hello') |
|
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
55 self.assertEqual(self.dex.find(['world']), [('test', '2', 'foo')]) |
|
a4edd24c32be
test fixes and checking of indexer overwrites (xapian currently fails)
Richard Jones <richard@users.sourceforge.net>
parents:
3297
diff
changeset
|
56 |
|
3547
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
57 def test_clear(self): |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
58 self.dex.add_text(('test', '1', 'foo'), 'a the hello world') |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
59 self.dex.add_text(('test', '2', 'foo'), 'blah blah the world') |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
60 self.assertEqual(self.dex.find(['world']), [('test', '1', 'foo'), |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
61 ('test', '2', 'foo')]) |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
62 self.dex.add_text(('test', '1', 'foo'), '') |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
63 self.assertEqual(self.dex.find(['world']), [('test', '2', 'foo')]) |
|
7728ee93efd2
fix reindexing in Xapian
Richard Jones <richard@users.sourceforge.net>
parents:
3546
diff
changeset
|
64 |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
65 def tearDown(self): |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
66 shutil.rmtree('test-index') |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
67 |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
68 class XapianIndexerTest(IndexerTest): |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
69 def setUp(self): |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
70 if os.path.exists('test-index'): |
|
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
71 shutil.rmtree('test-index') |
|
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
72 os.mkdir('test-index') |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
73 from roundup.backends.indexer_xapian import Indexer |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
74 self.dex = Indexer(db) |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
75 def tearDown(self): |
|
3297
8f7dc283bfa5
some more Xapian stuff (doc, test fixes)
Richard Jones <richard@users.sourceforge.net>
parents:
3295
diff
changeset
|
76 shutil.rmtree('test-index') |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
77 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
78 def test_suite(): |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
79 suite = unittest.TestSuite() |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
80 suite.addTest(unittest.makeSuite(IndexerTest)) |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
81 try: |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
82 import xapian |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
83 suite.addTest(unittest.makeSuite(XapianIndexerTest)) |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
84 except ImportError: |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
85 print "Skipping Xapian indexer tests" |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3078
diff
changeset
|
86 pass |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
87 return suite |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
88 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
89 if __name__ == '__main__': |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
90 runner = unittest.TextTestRunner() |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1090
diff
changeset
|
91 unittest.main(testRunner=runner) |
|
848
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
92 |
|
2a928d404af8
ehem, forgot to add
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
93 # vim: set filetype=python ts=4 sw=4 et si |
