Mercurial > p > roundup > code

annotate roundup/backends/back_sqlite.py @ 7402:bca4eaffb212

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Rename GNUMakefile to Makefile
author John Rouillard <rouilj@ieee.org>
date Wed, 24 May 2023 20:40:31 -0400
parents e138f5cf432a
children e10d0b262abd
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
1 """Implements a backend for SQLite.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3 See https://pysqlite.sourceforge.net/ for pysqlite info
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
4
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
5
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
6 NOTE: we use the rdbms_common table creation methods which define datatypes
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
7 for the columns, but sqlite IGNORES these specifications.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
8 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
9 __docformat__ = 'restructuredtext'
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
10
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
11 import logging
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
12 import os
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
13 import shutil
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
14 import time
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
15
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
16
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
17 from roundup import hyperdb, date, password
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
18 from roundup.backends import rdbms_common
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
19 from roundup.backends import sessions_sqlite
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
20 from roundup.backends import sessions_dbm
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
21
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
22 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
23 from roundup.backends import sessions_redis
6820
feb175271f3e redis import failure on python2 causes crash
John Rouillard <rouilj@ieee.org>
parents: 6814
diff changeset
24 except ImportError:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
25 sessions_redis = None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
26
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
27 from roundup.anypy.strings import uany2s
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
28
3728
b476fef16ccc fixed support for pysqlite2 (version 2.1.0 is the minimum version supported)
Richard Jones <richard@users.sourceforge.net>
parents: 3718
diff changeset
29 sqlite_version = None
3654
f5201bbbc505 update for latest version of pysqlite ([SF#1487098]; patch 1534227)
Richard Jones <richard@users.sourceforge.net>
parents: 3609
diff changeset
30 try:
4231
23ba17d01a24 re-order sqlite imports to handle multiple installed versions (issue 2550570)
Richard Jones <richard@users.sourceforge.net>
parents: 4091
diff changeset
31 import sqlite3 as sqlite
23ba17d01a24 re-order sqlite imports to handle multiple installed versions (issue 2550570)
Richard Jones <richard@users.sourceforge.net>
parents: 4091
diff changeset
32 sqlite_version = 3
3654
f5201bbbc505 update for latest version of pysqlite ([SF#1487098]; patch 1534227)
Richard Jones <richard@users.sourceforge.net>
parents: 3609
diff changeset
33 except ImportError:
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
34 try:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
35 from pysqlite2 import dbapi2 as sqlite
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
36 if sqlite.version_info < (2, 1, 0):
3794
f5ddd1639a15 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3752
diff changeset
37 raise ValueError('pysqlite2 minimum version is 2.1.0+ '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
38 '- %s found' % sqlite.version)
3728
b476fef16ccc fixed support for pysqlite2 (version 2.1.0 is the minimum version supported)
Richard Jones <richard@users.sourceforge.net>
parents: 3718
diff changeset
39 sqlite_version = 2
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
40 except ImportError:
4231
23ba17d01a24 re-order sqlite imports to handle multiple installed versions (issue 2550570)
Richard Jones <richard@users.sourceforge.net>
parents: 4091
diff changeset
41 import sqlite
23ba17d01a24 re-order sqlite imports to handle multiple installed versions (issue 2550570)
Richard Jones <richard@users.sourceforge.net>
parents: 4091
diff changeset
42 sqlite_version = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
43
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
44
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
45 def db_exists(config):
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
46 return os.path.exists(os.path.join(config.DATABASE, 'db'))
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
47
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
48
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
49 def db_nuke(config):
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
50 shutil.rmtree(config.DATABASE)
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
51
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
52
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
53 class Database(rdbms_common.Database):
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
54 """Sqlite DB backend implementation
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
55
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
56 attributes:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
57 dbtype:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
58 holds the value for the type of db. It is used by indexer to
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
59 identify the database type so it can import the correct indexer
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
60 module when using native text search mode.
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
61 """
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
62
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
63 # char to use for positional arguments
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
64 if sqlite_version in (2, 3):
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
65 arg = '?'
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
66 else:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
67 arg = '%s'
3048
d9b4224f955c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3008
diff changeset
68
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
69 dbtype = "sqlite"
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
70
3048
d9b4224f955c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3008
diff changeset
71 # used by some code to switch styles of query
d9b4224f955c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3008
diff changeset
72 implements_intersect = 1
d9b4224f955c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3008
diff changeset
73
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
74 # used in generic backend to determine if db supports
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
75 # 'DOUBLE PRECISION' for floating point numbers. Note that sqlite
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
76 # already has double precision as its standard 'REAL' type. So this
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
77 # is set to False here.
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
78
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
79 implements_double_precision = False
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5143
diff changeset
80
2168
7e595abb781e better sqlite columns
Richard Jones <richard@users.sourceforge.net>
parents: 2144
diff changeset
81 hyperdb_to_sql_datatypes = {
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
82 hyperdb.String : 'VARCHAR(255)', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
83 hyperdb.Date : 'VARCHAR(30)', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
84 hyperdb.Link : 'INTEGER', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
85 hyperdb.Interval : 'VARCHAR(255)', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
86 hyperdb.Password : 'VARCHAR(255)', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
87 hyperdb.Boolean : 'BOOLEAN', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
88 hyperdb.Number : 'REAL', # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
89 hyperdb.Integer : 'INTEGER', # noqa: E203
2168
7e595abb781e better sqlite columns
Richard Jones <richard@users.sourceforge.net>
parents: 2144
diff changeset
90 }
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
91 hyperdb_to_sql_value = {
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
92 hyperdb.String : str, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
93 hyperdb.Date : lambda x: x.serialise(), # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
94 hyperdb.Link : int, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
95 hyperdb.Interval : str, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
96 hyperdb.Password : str, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
97 hyperdb.Boolean : int, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
98 hyperdb.Integer : int, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
99 hyperdb.Number : lambda x: x, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
100 hyperdb.Multilink : lambda x: x, # used in journal marshalling, # noqa: E203
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
101 }
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
102 sql_to_hyperdb_value = {
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
103 hyperdb.String : uany2s, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
104 hyperdb.Date : lambda x: date.Date(str(x)), # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
105 hyperdb.Link : str, # XXX numeric ids # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
106 hyperdb.Interval : date.Interval, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
107 hyperdb.Password : lambda x: password.Password(encrypted=x), # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
108 hyperdb.Boolean : int, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
109 hyperdb.Integer : int, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
110 hyperdb.Number : rdbms_common._num_cvt, # noqa: E203
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
111 hyperdb.Multilink : lambda x: x, # used in journal marshalling, # noqa: E203
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
112 }
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
113
6933
e138f5cf432a fix comment on use of sqlite for session db's.
John Rouillard <rouilj@ieee.org>
parents: 6932
diff changeset
114 # We can use DBM, redis or SQLite for managing session info and
e138f5cf432a fix comment on use of sqlite for session db's.
John Rouillard <rouilj@ieee.org>
parents: 6932
diff changeset
115 # one-time keys:
e138f5cf432a fix comment on use of sqlite for session db's.
John Rouillard <rouilj@ieee.org>
parents: 6932
diff changeset
116 # For SQL database storage of this info we have to create separate
e138f5cf432a fix comment on use of sqlite for session db's.
John Rouillard <rouilj@ieee.org>
parents: 6932
diff changeset
117 # databases for Otk and Session because SQLite doesn't support
e138f5cf432a fix comment on use of sqlite for session db's.
John Rouillard <rouilj@ieee.org>
parents: 6932
diff changeset
118 # concurrent connections to the same database.
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
119 def getSessionManager(self):
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
120 if not self.Session:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
121 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
122 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
123 self.Session = sessions_sqlite.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
124 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
125 "redis module is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
126 self.Session = sessions_redis.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
127 elif self.config.SESSIONDB_BACKEND == "anydbm":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
128 self.Session = sessions_dbm.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
129 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
130 self.Session = sessions_sqlite.Sessions(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
131 return self.Session
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
132
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
133 def getOTKManager(self):
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
134 if not self.Otk:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
135 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
136 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
137 self.Session = sessions_sqlite.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
138 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
139 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
140 self.Otk = sessions_redis.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
141 elif self.config.SESSIONDB_BACKEND == "anydbm":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
142 self.Otk = sessions_dbm.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
143 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6804
diff changeset
144 self.Otk = sessions_sqlite.OneTimeKeys(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
145 return self.Otk
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
146
2847
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
147 def sqlite_busy_handler(self, data, table, count):
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
148 """invoked whenever SQLite tries to access a database that is locked"""
4415
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
149 now = time.time()
2847
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
150 if count == 1:
4415
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
151 # Timeout for handling locked database (default 30s)
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
152 self._busy_handler_endtime = now + self.config.RDBMS_SQLITE_TIMEOUT
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
153 elif now > self._busy_handler_endtime:
2847
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
154 # timeout expired - no more retries
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
155 return 0
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
156 # sleep adaptively as retry count grows,
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
157 # starting from about half a second
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
158 time_to_sleep = 0.01 * (2 << min(5, count))
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
159 time.sleep(time_to_sleep)
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
160 return 1
fddbbeb581bc sqlite_busy_timeout is not supported in pysqlite 1.1 (SQLite 3.x).
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2764
diff changeset
161
6804
25d08e15e3b4 issue2551224 - Replace dbm db for sessions/otks when using sqlite
John Rouillard <rouilj@ieee.org>
parents: 6599
diff changeset
162 def sql_open_connection(self, dbname=None):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
163 """Open a standard, non-autocommitting connection.
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
164
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
165 pysqlite will automatically BEGIN TRANSACTION for us.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
166 """
2764
d3b3f1b3d59e sqlite database creation requires existing directory.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2736
diff changeset
167 # make sure the database directory exists
d3b3f1b3d59e sqlite database creation requires existing directory.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2736
diff changeset
168 # database itself will be created by sqlite if needed
d3b3f1b3d59e sqlite database creation requires existing directory.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2736
diff changeset
169 if not os.path.isdir(self.config.DATABASE):
d3b3f1b3d59e sqlite database creation requires existing directory.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2736
diff changeset
170 os.makedirs(self.config.DATABASE)
d3b3f1b3d59e sqlite database creation requires existing directory.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2736
diff changeset
171
6804
25d08e15e3b4 issue2551224 - Replace dbm db for sessions/otks when using sqlite
John Rouillard <rouilj@ieee.org>
parents: 6599
diff changeset
172 if dbname:
25d08e15e3b4 issue2551224 - Replace dbm db for sessions/otks when using sqlite
John Rouillard <rouilj@ieee.org>
parents: 6599
diff changeset
173 db = os.path.join(self.config.DATABASE, 'db-' + dbname)
25d08e15e3b4 issue2551224 - Replace dbm db for sessions/otks when using sqlite
John Rouillard <rouilj@ieee.org>
parents: 6599
diff changeset
174 else:
25d08e15e3b4 issue2551224 - Replace dbm db for sessions/otks when using sqlite
John Rouillard <rouilj@ieee.org>
parents: 6599
diff changeset
175 db = os.path.join(self.config.DATABASE, 'db')
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
176 logging.getLogger('roundup.hyperdb').info('open database %r' % db)
4415
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
177 # set timeout (30 second default is extraordinarily generous)
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
178 # for handling locked database
3728
b476fef16ccc fixed support for pysqlite2 (version 2.1.0 is the minimum version supported)
Richard Jones <richard@users.sourceforge.net>
parents: 3718
diff changeset
179 if sqlite_version == 1:
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
180 conn = sqlite.connect(db=db)
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
181 conn.db.sqlite_busy_handler(self.sqlite_busy_handler)
3728
b476fef16ccc fixed support for pysqlite2 (version 2.1.0 is the minimum version supported)
Richard Jones <richard@users.sourceforge.net>
parents: 3718
diff changeset
182 else:
4415
3e35233ea93c new rdbms config item sqlite_timeout...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
183 conn = sqlite.connect(db, timeout=self.config.RDBMS_SQLITE_TIMEOUT)
3728
b476fef16ccc fixed support for pysqlite2 (version 2.1.0 is the minimum version supported)
Richard Jones <richard@users.sourceforge.net>
parents: 3718
diff changeset
184 conn.row_factory = sqlite.Row
4091
09e79cbeb827 force sqlite3 in py2.6+ to treat our stored text as UTF-8
Richard Jones <richard@users.sourceforge.net>
parents: 4076
diff changeset
185
4240
289f249ba192 compatibilty fix for pysqlite2 / unicode
Richard Jones <richard@users.sourceforge.net>
parents: 4231
diff changeset
186 # pysqlite2 / sqlite3 want us to store Unicode in the db but
289f249ba192 compatibilty fix for pysqlite2 / unicode
Richard Jones <richard@users.sourceforge.net>
parents: 4231
diff changeset
187 # that's not what's been done historically and it's definitely
289f249ba192 compatibilty fix for pysqlite2 / unicode
Richard Jones <richard@users.sourceforge.net>
parents: 4231
diff changeset
188 # not what the other backends do, so we'll stick with UTF-8
289f249ba192 compatibilty fix for pysqlite2 / unicode
Richard Jones <richard@users.sourceforge.net>
parents: 4231
diff changeset
189 if sqlite_version in (2, 3):
4091
09e79cbeb827 force sqlite3 in py2.6+ to treat our stored text as UTF-8
Richard Jones <richard@users.sourceforge.net>
parents: 4076
diff changeset
190 conn.text_factory = str
09e79cbeb827 force sqlite3 in py2.6+ to treat our stored text as UTF-8
Richard Jones <richard@users.sourceforge.net>
parents: 4076
diff changeset
191
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
192 cursor = conn.cursor()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
193 return (conn, cursor)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
194
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
195 def open_connection(self):
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
196 # ensure files are group readable and writable
3609
f2fda3e6fc8b umask is now configurable (with the same 0002 default)
Richard Jones <richard@users.sourceforge.net>
parents: 3347
diff changeset
197 os.umask(self.config.UMASK)
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
198
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
199 (self.conn, self.cursor) = self.sql_open_connection()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
200
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 try:
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
202 self.load_dbschema()
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5175
diff changeset
203 except sqlite.DatabaseError as error:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 if str(error) != 'no such table: schema':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
205 raise
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
206 self.init_dbschema()
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
207 self.sql('create table schema (schema varchar)')
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
208 self.sql('create table ids (name varchar, num integer)')
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
209 self.sql('create index ids_name_idx on ids(name)')
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
210 self.create_version_2_tables()
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
211 self._add_fts5_table()
6917
fba76e0bba98 set journal mode to wal when creating db.
John Rouillard <rouilj@ieee.org>
parents: 6916
diff changeset
212 # Set journal mode to WAL.
6930
a96a239db0d9 Set all sqlite db's to WAL mode on creation
John Rouillard <rouilj@ieee.org>
parents: 6917
diff changeset
213 self.sql_commit() # close out rollback journal/transaction
a96a239db0d9 Set all sqlite db's to WAL mode on creation
John Rouillard <rouilj@ieee.org>
parents: 6917
diff changeset
214 self.sql('pragma journal_mode=wal') # set wal
a96a239db0d9 Set all sqlite db's to WAL mode on creation
John Rouillard <rouilj@ieee.org>
parents: 6917
diff changeset
215 self.sql_commit() # close out rollback and commit wal change
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
216
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
217 def create_version_2_tables(self):
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
218 self.sql('create table otks (otk_key varchar, '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
219 'otk_value varchar, otk_time integer)')
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
220 self.sql('create index otks_key_idx on otks(otk_key)')
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
221 self.sql('create table sessions (session_key varchar, '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
222 'session_time integer, session_value varchar)')
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
223 self.sql('create index sessions_key_idx on '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
224 'sessions(session_key)')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
225
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
226 # full-text indexing store
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
227 self.sql('CREATE TABLE __textids (_class varchar, '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
228 '_itemid varchar, _prop varchar, _textid'
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
229 ' integer primary key) ')
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
230 self.sql('CREATE TABLE __words (_word varchar, '
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
231 '_textid integer)')
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
232 self.sql('CREATE INDEX words_word_ids ON __words(_word)')
3858
bb30bbfc7cdd Indexing fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3794
diff changeset
233 self.sql('CREATE INDEX words_by_id ON __words (_textid)')
bb30bbfc7cdd Indexing fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3794
diff changeset
234 self.sql('CREATE UNIQUE INDEX __textids_by_props ON '
bb30bbfc7cdd Indexing fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3794
diff changeset
235 '__textids (_class, _itemid, _prop)')
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
236 sql = 'insert into ids (name, num) values (%s,%s)' % (
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
237 self.arg, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
238 self.sql(sql, ('__textids', 1))
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
239
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
240 def add_new_columns_v2(self):
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
241 # update existing tables to have the new actor column
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
242 tables = self.database_schema['tables']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
243 for classname, spec in self.classes.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
244 if classname in tables:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
245 dbspec = tables[classname]
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
246 self.update_class(spec, dbspec, force=1, adding_v2=1)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
247 # we've updated - don't try again
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
248 tables[classname] = spec.schema()
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
249
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2645
diff changeset
250 def fix_version_3_tables(self):
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2645
diff changeset
251 # NOOP - no restriction on column length here
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2645
diff changeset
252 pass
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2645
diff changeset
253
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
254 def _add_fts5_table(self):
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
255 self.sql('CREATE virtual TABLE __fts USING fts5(_class, '
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
256 '_itemid, _prop, _textblob)')
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
257
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
258 def fix_version_6_tables(self):
6599
39189dd94f2c issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents: 6588
diff changeset
259 # note sqlite has no limit on column size so v6 fixes
39189dd94f2c issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents: 6588
diff changeset
260 # to __words._word length are not needed.
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
261 # Add native full-text indexing table
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
262 self._add_fts5_table()
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6050
diff changeset
263
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
264 def update_class(self, spec, old_spec, force=0, adding_v2=0):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
265 """ Determine the differences between the current spec and the
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
266 database version of the spec, and update where necessary.
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
267
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
268 If 'force' is true, update the database anyway.
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
269
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
270 SQLite doesn't have ALTER TABLE, so we have to copy and
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
271 regenerate the tables with the new schema.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
272 """
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
273 new_spec = spec.schema()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
274 new_spec[1].sort()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
275 old_spec[1].sort()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
276 if not force and new_spec == old_spec:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
277 # no changes
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
278 return 0
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
279
4421
67bef70ab9b9 - more logger fixes, sorry for the noise.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4420
diff changeset
280 logging.getLogger('roundup.hyperdb').info(
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
281 'update_class %s' % spec.classname)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
282
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
283 # detect multilinks that have been removed, and drop their table
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
284 old_has = {}
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
285 for name, prop in old_spec[1]:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
286 old_has[name] = 1
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
287 if name in spec.properties or not \
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
288 isinstance(prop, hyperdb.Multilink):
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
289 continue
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
290 # it's a multilink, and it's been removed - drop the old
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
291 # table. First drop indexes.
3347
9020fe58c51c more errors. backport candidate
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 3310
diff changeset
292 self.drop_multilink_table_indexes(spec.classname, name)
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
293 sql = 'drop table %s_%s' % (spec.classname, prop)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
294 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
295
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
296 # now figure how we populate the new table
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
297 if adding_v2:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
298 fetch = ['_activity', '_creation', '_creator']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
299 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
300 fetch = ['_actor', '_activity', '_creation', '_creator']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
301 properties = spec.getprops()
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
302 for propname, _x in new_spec[1]:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
303 prop = properties[propname]
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
304 if isinstance(prop, hyperdb.Multilink):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
305 if propname not in old_has:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
306 # we need to create the new table
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
307 self.create_multilink_table(spec, propname)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
308 elif force:
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
309 tn = '%s_%s' % (spec.classname, propname)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
310 # grabe the current values
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
311 sql = 'select linkid, nodeid from %s' % tn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
312 self.sql(sql)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
313 rows = self.cursor.fetchall()
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
314
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
315 # drop the old table
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
316 self.drop_multilink_table_indexes(spec.classname, propname)
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
317 sql = 'drop table %s' % tn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
318 self.sql(sql)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
319
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
320 # re-create and populate the new table
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
321 self.create_multilink_table(spec, propname)
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
322 sql = """insert into %s (linkid, nodeid) values
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
323 (%s, %s)""" % (tn, self.arg, self.arg)
2137
c49495585c44 Fix for sqlite backend migration. Change Cookie -> SimpleCookie
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
324 for linkid, nodeid in rows:
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
325 self.sql(sql, (int(linkid), int(nodeid)))
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
326 elif propname in old_has:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
327 # we copy this col over from the old table
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
328 fetch.append('_'+propname)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
329
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
330 # select the data out of the old table
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
331 fetch.append('id')
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
332 fetch.append('__retired__')
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
333 fetchcols = ','.join(fetch)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
334 cn = spec.classname
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
335 sql = 'select %s from _%s' % (fetchcols, cn)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
336 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
337 olddata = self.cursor.fetchall()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
338
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
339 # TODO: update all the other index dropping code
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
340 self.drop_class_table_indexes(cn, old_spec[0])
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
341
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
342 # drop the old table
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
343 self.sql('drop table _%s' % cn)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
344
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
345 # create the new table
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
346 self.create_class_table(spec)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
347
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
348 if olddata:
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
349 inscols = ['id', '_actor', '_activity', '_creation',
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
350 '_creator', '__retired__']
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
351 for propname, _x in new_spec[1]:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
352 prop = properties[propname]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
353 if isinstance(prop, hyperdb.Multilink):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
354 continue
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
355 elif isinstance(prop, hyperdb.Interval):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
356 inscols.append('_'+propname)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
357 inscols.append('__'+propname+'_int__')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
358 elif propname in old_has:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
359 # we copy this col over from the old table
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
360 inscols.append('_'+propname)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
361
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
362 # do the insert of the old data - the new columns will have
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
363 # NULL values
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
364 args = ','.join([self.arg for x in inscols])
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
365 cols = ','.join(inscols)
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
366 sql = 'insert into _%s (%s) values (%s)' % (cn, cols, args)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
367 for entry in olddata:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
368 d = []
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4421
diff changeset
369 retired_id = None
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
370 for name in inscols:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
371 # generate the new value for the Interval int column
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
372 if name.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
373 name = name[2:-6]
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
374 if sqlite_version in (2, 3):
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
375 try:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
376 v = hyperdb.Interval(entry[name]).as_seconds()
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
377 except IndexError:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
378 v = None
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
379 elif name in entry:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
380 v = hyperdb.Interval(entry[name]).as_seconds()
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
381 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
382 v = None
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
383 elif sqlite_version in (2, 3):
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
384 try:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
385 v = entry[name]
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
386 except IndexError:
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
387 v = None
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4240
diff changeset
388 elif (sqlite_version == 1 and name in entry):
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
389 v = entry[name]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
390 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
391 v = None
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4421
diff changeset
392 if name == 'id':
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4421
diff changeset
393 retired_id = v
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
394 elif (name == '__retired__' and retired_id and
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
395 v not in ['0', 0]):
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4421
diff changeset
396 v = retired_id
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
397 d.append(v)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
398 self.sql(sql, tuple(d))
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
399
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
400 return 1
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
401
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
402 def sql_close(self):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
403 """ Squash any error caused by us already having closed the
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
404 connection.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
405 """
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
406 try:
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
407 self.conn.close()
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5175
diff changeset
408 except sqlite.ProgrammingError as value:
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
409 if str(value) != 'close failed - Connection is closed.':
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
410 raise
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
411
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
412 def sql_rollback(self):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
413 """ Squash any error caused by us having closed the connection (and
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
414 therefore not having anything to roll back)
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
415 """
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
416 try:
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
417 self.conn.rollback()
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5175
diff changeset
418 except sqlite.ProgrammingError as value:
1236
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
419 if str(value) != 'rollback failed - Connection is closed.':
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
420 raise
dd52bf10f934 Bug fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
421
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
422 def __repr__(self):
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
423 return '<roundlite 0x%x>' % id(self)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
424
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
425 def sql_commit(self):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
426 """ Actually commit to the database.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
427
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
428 Ignore errors if there's nothing to commit.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
429 """
6916
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
430 def list_dir(dir):
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
431 import os
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
432 files = os.listdir(self.dir)
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
433 # ['db-journal', 'files', 'db']
6916
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
434 for entry in [''] + files:
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
435 path = self.dir + '/' + entry
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
436 stat = os.stat(path)
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
437 print("file: %s, uid: %s, gid: %s, mode: %o" % (
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
438 path, stat.st_uid, stat.st_gid, stat.st_mode))
6916
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
439
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
440 # Getting sqlite3.OperationalError: disk I/O error
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
441 # in CI. It happens intermittently. Try to get more
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
442 # info about what is happening and retry the commit.
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
443 # Some possibilities:
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
444 # -journal file not writable
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
445 # file has disappeared
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
446 #
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
447 # Note after exception self.conn.in_transaction is False
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
448 # but was True before failed commit(). Retry succeeds,
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
449 # but I am not sure it actually does anything.
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
450 # for retry in range(2):
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
451 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
452 self.conn.commit()
6916
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
453 except sqlite.OperationalError as error:
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
454 if str(error) != 'disk I/O error':
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
455 raise
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
456 list_dir(self.dir)
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
457 raise
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5175
diff changeset
458 except sqlite.DatabaseError as error:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
459 if str(error) != 'cannot commit - no transaction is active':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
460 raise
6916
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
461 # else:
fad6b3a272cd Add instrumentation to debug I/O error in CI that I don't get locally
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
462 # break # out of loop if no exception
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
463 # open a new cursor for subsequent work
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2273
diff changeset
464 self.cursor = self.conn.cursor()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
465
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
466 def sql_index_exists(self, table_name, index_name):
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
467 self.sql('pragma index_list(%s)' % table_name)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
468 for entry in self.cursor.fetchall():
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
469 if entry[1] == index_name:
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
470 return 1
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
471 return 0
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1836
diff changeset
472
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
473 # old-skool id generation
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
474 def newid(self, classname):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
475 """ Generate a new id for the given class
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
476 """
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
477
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
478 # Prevent other processes from reading while we increment.
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
479 # Otherwise multiple processes can end up with the same
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
480 # new id and hilarity results.
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
481 #
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
482 # Defeat pysqlite's attempts to do locking by setting
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
483 # isolation_level to None. Pysqlite can commit
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
484 # on it's own even if we don't want it to end the transaction.
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
485 # If we rewrite to use another sqlite library like apsw we
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
486 # don't have to deal with this autocommit/autotransact foolishness.
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
487 self.conn.isolation_level = None
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
488 # Manage the transaction locks manually.
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
489 self.sql("BEGIN IMMEDIATE")
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
490
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
491 # get the next ID
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
492 sql = 'select num from ids where name=%s' % self.arg
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
493 self.sql(sql, (classname, ))
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
494 newid = int(self.cursor.fetchone()[0])
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
495
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
496 # leave the next larger number as the next newid
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
497 sql = 'update ids set num=num+1 where name=%s' % self.arg
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
498 vals = (classname,)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
499 self.sql(sql, vals)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
500
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
501 # reset pysqlite's auto transact stuff to default since the
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
502 # rest of the code expects it.
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
503 self.conn.isolation_level = ''
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
504 # commit writing the data, clearing locks for other processes
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
505 # and create a new cursor to the database.
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
506 self.sql_commit()
5143
9cebf686c552 issue2550727: db.newid is broken with sqlite.
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
507
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
508 # return as string
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
509 return str(newid)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
510
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
511 def setid(self, classname, setid):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
512 """ Set the id counter: used during import of database
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
513
3306
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
514 We add one to make it behave like the sequences in postgres.
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
515 """
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
516 sql = 'update ids set num=%s where name=%s' % (self.arg, self.arg)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
517 vals = (int(setid)+1, classname)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
518 self.sql(sql, vals)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
519
3306
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
520 def clear(self):
3310
3518d1ffd940 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3306
diff changeset
521 rdbms_common.Database.clear(self)
3306
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
522 # set the id counters to 0 (setid adds one) so we start at 1
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
523 for cn in self.classes.keys():
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
524 self.setid(cn, 0)
ae1093684ef5 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 3163
diff changeset
525
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
526 def create_class(self, spec):
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
527 rdbms_common.Database.create_class(self, spec)
6932
ff7f6c0bbdae flake8 fixes.
John Rouillard <rouilj@ieee.org>
parents: 6930
diff changeset
528 sql = 'insert into ids (name, num) values (%s, %s)' % (
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
529 self.arg, self.arg)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
530 vals = (spec.classname, 1)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
531 self.sql(sql, vals)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2094
diff changeset
532
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
533 if sqlite_version in (2, 3):
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
534 def load_journal(self, classname, cols, nodeid):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
535 """We need to turn the sqlite3.Row into a tuple so it can be
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
536 unpacked"""
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
537 l = rdbms_common.Database.load_journal(self,
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
538 classname, cols, nodeid)
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
539 cols = range(5)
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
540 return [[row[col] for col in cols] for row in l]
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3693
diff changeset
541
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
542
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
543 class sqliteClass:
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
544 def filter(self, *args, **kw):
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
545 """ If there's NO matches to a fetch, sqlite returns NULL
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
546 instead of nothing
4076
59d02edfaa9c Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 3858
diff changeset
547 """
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
548 return [f for f in rdbms_common.Class.filter(self, *args, **kw) if f]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
549
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
550
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
551 class Class(sqliteClass, rdbms_common.Class):
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
552 pass
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
553
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
554
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
555 class IssueClass(sqliteClass, rdbms_common.IssueClass):
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
556 pass
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
557
6050
19a58adf5d1f Flake8: whitepace; don't import base64; unused loop var gets leading _
John Rouillard <rouilj@ieee.org>
parents: 5869
diff changeset
558
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
559 class FileClass(sqliteClass, rdbms_common.FileClass):
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
560 pass
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
561
2645
8250c63c3963 remove space at EOL, add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2633
diff changeset
562 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/