Mercurial > p > roundup > code

annotate roundup/roundup_indexer.py @ 703:8d2cb0d09da4 search_indexing-0-4-2-branch

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed small bug that prevented indexes from being generated.
author Roche Compaan <rochecompaan@users.sourceforge.net>
date Thu, 02 May 2002 11:52:12 +0000
parents 0521ddc7bb31
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
681
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
1 #
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
6 #
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
11 #
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
17 #
703
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
18 #$Id: roundup_indexer.py,v 1.1.2.3 2002-05-02 11:52:12 rochecompaan Exp $
681
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
19 '''
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
20 This module provides an indexer class, RoundupIndexer, that stores text
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
21 indices in a roundup instance. This class makes searching the content of
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
22 messages and text files possible.
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
23 '''
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
24 import os
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
25 from roundup.indexer import SlicedZPickleIndexer
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
26
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
27 class RoundupIndexer(SlicedZPickleIndexer):
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
28 ''' Indexes messages and files
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
29 '''
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
30
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
31 def __init__(self, db_path):
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
32 indexdb_path = os.path.join(db_path, 'indexes')
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
33 index_exists = 0
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
34 if not os.path.exists(indexdb_path):
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
35 os.makedirs(indexdb_path)
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
36 os.chmod(indexdb_path, 0775)
703
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
37 else:
681
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
38 index_exists = 1
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
39 index_path = os.path.join(indexdb_path, 'index.db')
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
40 SlicedZPickleIndexer.__init__(self,
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
41 INDEXDB=index_path, QUIET=9)
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
42 files_path = os.path.join(db_path, 'files')
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
43 if not index_exists:
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
44 self.add_files(dir=files_path)
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
45 self.save_index()
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
46
692
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
47 def search(self, search_terms, klass):
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
48 ''' display search results
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
49 '''
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
50 hits = self.find(search_terms)
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
51 links = []
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
52 nodeids = {}
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
53 designator_propname = {'msg': 'messages',
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
54 'file': 'files'}
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
55 if hits:
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
56 hitcount = len(hits)
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
57 # build a dictionary of nodes and their associated messages
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
58 # and files
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
59 for hit in hits.keys():
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
60 filename = hits[hit].split('/')[-1]
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
61 for designator, propname in designator_propname.items():
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
62 if filename.find(designator) == -1: continue
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
63 nodeid = filename[len(designator):]
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
64 result = apply(klass.find, (), {propname:nodeid})
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
65 if not result: continue
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
66
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
67 id = str(result[0])
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
68 if not nodeids.has_key(id):
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
69 nodeids[id] = {}
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
70
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
71 node_dict = nodeids[id]
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
72 if not node_dict.has_key(propname):
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
73 node_dict[propname] = [nodeid]
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
74 elif node_dict.has_key(propname):
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
75 node_dict[propname].append(nodeid)
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
76
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
77 return nodeids
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
78
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
79
681
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
80 #
1b2d0e702ca8 Added feature [SF#526730] - search for messages capability
Roche Compaan <rochecompaan@users.sourceforge.net>
parents:
diff changeset
81 #$Log: not supported by cvs2svn $
703
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
82 #Revision 1.1.2.2 2002/04/19 19:54:42 rochecompaan
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
83 #cgi_client.py
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
84 # removed search link for the time being
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
85 # moved rendering of matches to htmltemplate
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
86 #hyperdb.py
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
87 # filtering of nodes on full text search incorporated in filter method
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
88 #roundupdb.py
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
89 # added paramater to call of filter method
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
90 #roundup_indexer.py
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
91 # added search method to RoundupIndexer class
8d2cb0d09da4 Fixed small bug that prevented indexes from being generated.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 692
diff changeset
92 #
692
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
93 #Revision 1.1.2.1 2002/04/03 11:55:57 rochecompaan
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
94 # . Added feature #526730 - search for messages capability
0521ddc7bb31 Miscellaneous changes.
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 681
diff changeset
95 #
Roundup Issue Tracker: http://roundup-tracker.org/