Mercurial > p > roundup > code

annotate roundup/mlink_expr.py @ 6610:db3f0ba75b4a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Change checkpoint_data and restore_connection_on_error to subtransaction checkpoint_data and restore_connection_on_error used to commit() and rollback() the db connection. This causes additional I/O and load. Changed them to use 'SAVEPOINT name' and 'ROLLBACK TO name' to get a faster method for handling errors within a tranaction. One thing to note is that postgresql (unlike SQL std) doesn't overwrite an older savepoint with he same name. It keeps all savepoints but only rolls back to the newest one with a given name. This could be a resource issue. I left a commented out release statement in case somebody runs into an issue due to too many savepoints. I expect it to slow down the import but....
author John Rouillard <rouilj@ieee.org>
date Sat, 29 Jan 2022 11:29:36 -0500
parents a0c0ee3ed8b1
children 7b090bb35c25
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
1 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
2 # Copyright: 2010 Intevation GmbH.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
3 # 2021 Ralf Schlatterbeck, rsc@runtux.com.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
4 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
5
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
6 # This module is Free Software under the Roundup licensing,
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
7 # see the COPYING.txt file coming with Roundup.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
8 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
9
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
10 class Binary:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
11
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
12 def __init__(self, x, y):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
13 self.x = x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
14 self.y = y
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
15
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
16 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
17 self.x.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
18 self.y.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
19
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
20 class Unary:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
21
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
22 def __init__(self, x):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
23 self.x = x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
24
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
25 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
26 return atom(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
27
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
28 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
29 self.x.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
30
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
31 class Equals(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
32
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
33 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
34 return self.x in v
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
35
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
36 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
37 visitor(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
38
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
39 class Empty(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
40
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
41 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
42 return not v
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
43
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
44 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
45 visitor(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
46
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
47 class Not(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
48
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
49 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
50 return not self.x.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
51
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
52 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
53 return "NOT(%s)" % self.x.generate(atom)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
54
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
55 class Or(Binary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
56
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
57 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
58 return self.x.evaluate(v) or self.y.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
59
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
60 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
61 return "(%s)OR(%s)" % (
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
62 self.x.generate(atom),
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
63 self.y.generate(atom))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
64
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
65 class And(Binary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
66
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
67 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
68 return self.x.evaluate(v) and self.y.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
69
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
70 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
71 return "(%s)AND(%s)" % (
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
72 self.x.generate(atom),
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
73 self.y.generate(atom))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
74
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
75 def compile_expression(opcodes):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
76
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
77 stack = []
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
78 push, pop = stack.append, stack.pop
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
79 for opcode in opcodes:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
80 if opcode == -1: push(Empty(opcode))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
81 elif opcode == -2: push(Not(pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
82 elif opcode == -3: push(And(pop(), pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
83 elif opcode == -4: push(Or(pop(), pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
84 else: push(Equals(opcode))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
85
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
86 return pop()
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
87
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
88 class Expression:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
89
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
90 def __init__(self, v, is_link=False):
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
91 try:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
92 opcodes = [int(x) for x in v]
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
93 if min(opcodes) >= -1:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
94 raise ValueError()
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
95
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
96 compiled = compile_expression(opcodes)
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
97 if is_link:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
98 self.evaluate = lambda x: compiled.evaluate(
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
99 x and [int(x)] or [])
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
100 else:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
101 self.evaluate = lambda x: compiled.evaluate([int(y) for y in x])
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
102 except:
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
103 if is_link:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
104 v = [None if x == '-1' else x for x in v]
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
105 self.evaluate = lambda x: x in v
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
106 elif '-1' in v:
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
107 v = [x for x in v if int(x) > 0]
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
108 self.evaluate = lambda x: bool(set(x) & set(v)) or not x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
109 else:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
110 self.evaluate = lambda x: bool(set(x) & set(v))
Roundup Issue Tracker: http://roundup-tracker.org/