Mercurial > p > roundup > code

annotate roundup/roundup_indexer.py @ 752:a721f4e7ebbc

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Installation note for people running the tests with a CVS checkout.
author Richard Jones <richard@users.sourceforge.net>
date Tue, 28 May 2002 11:52:08 +0000
parents 51c425129b35
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
749
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
1 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
6 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
11 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
17 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
18 #$Id: roundup_indexer.py,v 1.2 2002-05-25 07:16:24 rochecompaan Exp $
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
19 '''
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
20 This module provides an indexer class, RoundupIndexer, that stores text
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
21 indices in a roundup instance. This class makes searching the content of
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
22 messages and text files possible.
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
23 '''
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
24 import os
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
25 from roundup.indexer import SlicedZPickleIndexer
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
26
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
27 class RoundupIndexer(SlicedZPickleIndexer):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
28 ''' Indexes messages and files
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
29 '''
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
30
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
31 def __init__(self, db_path):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
32 indexdb_path = os.path.join(db_path, 'indexes')
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
33 index_exists = 0
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
34 if not os.path.exists(indexdb_path):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
35 os.makedirs(indexdb_path)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
36 os.chmod(indexdb_path, 0775)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
37 else:
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
38 index_exists = 1
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
39 index_path = os.path.join(indexdb_path, 'index.db')
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
40 SlicedZPickleIndexer.__init__(self,
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
41 INDEXDB=index_path, QUIET=9)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
42 files_path = os.path.join(db_path, 'files')
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
43 if not index_exists:
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
44 self.add_files(dir=files_path)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
45 self.save_index()
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
46
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
47 def search(self, search_terms, klass):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
48 ''' display search results
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
49 '''
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
50 hits = self.find(search_terms)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
51 links = []
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
52 nodeids = {}
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
53 designator_propname = {'msg': 'messages',
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
54 'file': 'files'}
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
55 if hits:
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
56 hitcount = len(hits)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
57 # build a dictionary of nodes and their associated messages
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
58 # and files
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
59 for hit in hits.keys():
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
60 filename = hits[hit].split('/')[-1]
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
61 for designator, propname in designator_propname.items():
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
62 if filename.find(designator) == -1: continue
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
63 nodeid = filename[len(designator):]
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
64 result = apply(klass.find, (), {propname:nodeid})
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
65 if not result: continue
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
66
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
67 id = str(result[0])
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
68 if not nodeids.has_key(id):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
69 nodeids[id] = {}
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
70
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
71 node_dict = nodeids[id]
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
72 if not node_dict.has_key(propname):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
73 node_dict[propname] = [nodeid]
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
74 elif node_dict.has_key(propname):
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
75 node_dict[propname].append(nodeid)
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
76
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
77 return nodeids
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
78
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
79
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
80 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
81 #$Log: not supported by cvs2svn $
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
82 #Revision 1.1.2.3 2002/05/02 11:52:12 rochecompaan
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
83 #Fixed small bug that prevented indexes from being generated.
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
84 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
85 #Revision 1.1.2.2 2002/04/19 19:54:42 rochecompaan
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
86 #cgi_client.py
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
87 # removed search link for the time being
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
88 # moved rendering of matches to htmltemplate
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
89 #hyperdb.py
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
90 # filtering of nodes on full text search incorporated in filter method
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
91 #roundupdb.py
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
92 # added paramater to call of filter method
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
93 #roundup_indexer.py
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
94 # added search method to RoundupIndexer class
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
95 #
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
96 #Revision 1.1.2.1 2002/04/03 11:55:57 rochecompaan
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
97 # . Added feature #526730 - search for messages capability
51c425129b35 Merged search_indexing-branch with HEAD
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
98 #
Roundup Issue Tracker: http://roundup-tracker.org/