annotate roundup/mlink_expr.py @ 6433:c1d3fbcdbfbd

issue2551142 - Import of retired node ... unique constraint failure. Title: Import of retired node with username after active node fails with unique constraint failure. More fixes needed for mysql and postgresql. mysql: add unique constraint for (keyvalue, __retired__) when creating class in the database. On schema change if class is changed, remove the unique constraint too. upgrade version of rdbms database from 5 to 6 to add constraint to all version 5 databases that were created as version 5 and didn't get the unique constraint. Make no changes on version 5 databases upgraded from version 4, the upgrade process to 5 added the constraint. Make no changes to other databases (sqlite, postgres) during upgrade from version 5 to 6. postgres: Handle the exception raised on unique constraint violation. The exception invalidates the database connection so it can't be used to recover from the exception. Added two new database methods: checkpoint_data - performs a db.commit under postgres does nothing on other backends restore_connection_on_error - does a db.rollback on postgres, does nothing on other backends with the rollback() done on the connection I can use the database connection to fixup the import that failed on the unique constraint. This makes postgres slower but without the commit after every imported object, the rollback will delete all the entries done up to this point. Trying to figure out how to make the caller do_import batch and recover from this failure is beyond me. Also dismissed having to process the export csv file before importing. Pushing that onto a user just seems wrong. Also since import/export isn't frequently done the lack of surprise on having a failing import and reduced load/frustration for the user seems worth it. Also the import can be run in verbose mode where it prints out a row as it is processed, so it may take a while, ut the user can get feedback. db_test-base.py: add test for upgrade from 5 to 6.
author John Rouillard <rouilj@ieee.org>
date Thu, 10 Jun 2021 12:52:05 -0400
parents a0c0ee3ed8b1
children 7b090bb35c25
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
1 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
2 # Copyright: 2010 Intevation GmbH.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
3 # 2021 Ralf Schlatterbeck, rsc@runtux.com.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
4 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
5
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
6 # This module is Free Software under the Roundup licensing,
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
7 # see the COPYING.txt file coming with Roundup.
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
8 #
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
9
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
10 class Binary:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
11
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
12 def __init__(self, x, y):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
13 self.x = x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
14 self.y = y
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
15
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
16 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
17 self.x.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
18 self.y.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
19
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
20 class Unary:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
21
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
22 def __init__(self, x):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
23 self.x = x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
24
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
25 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
26 return atom(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
27
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
28 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
29 self.x.visit(visitor)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
30
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
31 class Equals(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
32
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
33 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
34 return self.x in v
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
35
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
36 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
37 visitor(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
38
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
39 class Empty(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
40
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
41 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
42 return not v
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
43
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
44 def visit(self, visitor):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
45 visitor(self)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
46
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
47 class Not(Unary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
48
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
49 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
50 return not self.x.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
51
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
52 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
53 return "NOT(%s)" % self.x.generate(atom)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
54
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
55 class Or(Binary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
56
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
57 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
58 return self.x.evaluate(v) or self.y.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
59
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
60 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
61 return "(%s)OR(%s)" % (
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
62 self.x.generate(atom),
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
63 self.y.generate(atom))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
64
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
65 class And(Binary):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
66
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
67 def evaluate(self, v):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
68 return self.x.evaluate(v) and self.y.evaluate(v)
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
69
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
70 def generate(self, atom):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
71 return "(%s)AND(%s)" % (
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
72 self.x.generate(atom),
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
73 self.y.generate(atom))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
74
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
75 def compile_expression(opcodes):
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
76
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
77 stack = []
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
78 push, pop = stack.append, stack.pop
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
79 for opcode in opcodes:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
80 if opcode == -1: push(Empty(opcode))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
81 elif opcode == -2: push(Not(pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
82 elif opcode == -3: push(And(pop(), pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
83 elif opcode == -4: push(Or(pop(), pop()))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
84 else: push(Equals(opcode))
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
85
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
86 return pop()
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
87
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
88 class Expression:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
89
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
90 def __init__(self, v, is_link=False):
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
91 try:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
92 opcodes = [int(x) for x in v]
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
93 if min(opcodes) >= -1:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
94 raise ValueError()
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
95
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
96 compiled = compile_expression(opcodes)
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
97 if is_link:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
98 self.evaluate = lambda x: compiled.evaluate(
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
99 x and [int(x)] or [])
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
100 else:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
101 self.evaluate = lambda x: compiled.evaluate([int(y) for y in x])
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
102 except:
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
103 if is_link:
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
104 v = [None if x == '-1' else x for x in v]
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
105 self.evaluate = lambda x: x in v
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
106 elif '-1' in v:
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
107 v = [x for x in v if int(x) > 0]
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
108 self.evaluate = lambda x: bool(set(x) & set(v)) or not x
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
109 else:
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
diff changeset
110 self.evaluate = lambda x: bool(set(x) & set(v))

Roundup Issue Tracker: http://roundup-tracker.org/