Mercurial > p > roundup > code

annotate roundup/backends/blobfiles.py @ 6936:16d6d1f9038a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
flake8 fixes
author John Rouillard <rouilj@ieee.org>
date Thu, 08 Sep 2022 13:50:36 -0400
parents 8c371af7e785
children 1045425c23b2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
1 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
6 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
11 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
17 #
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
18 """This module exports file storage for roundup backends.
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
19 Files are stored into a directory hierarchy.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
20 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1946
diff changeset
21 __docformat__ = 'restructuredtext'
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
22
818
254b8d112eec cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
23 import os
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
24
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
25
2496
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
26 def files_in_dir(dir):
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
27 if not os.path.exists(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
28 return 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
29 num_files = 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
30 for dir_entry in os.listdir(dir):
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
31 full_filename = os.path.join(dir, dir_entry)
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
32 if os.path.isfile(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
33 num_files = num_files + 1
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
34 elif os.path.isdir(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
35 num_files = num_files + files_in_dir(full_filename)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
36 return num_files
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
37
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
38
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
39 class FileStorage(object):
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
40 """Store files in some directory structure
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
41
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
42 Some databases do not permit the storage of arbitrary data (i.e.,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
43 file content). And, some database schema explicitly store file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
44 content in the fielsystem. In particular, if a class defines a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
45 'filename' property, it is assumed that the data is stored in the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
46 indicated file, outside of whatever database Roundup is otherwise
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
47 using.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
48
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
49 In these situations, it is difficult to maintain the transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
50 abstractions used elsewhere in Roundup. In particular, if a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
51 file's content is edited, but then the containing transaction is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
52 not committed, we do not want to commit the edit. Similarly, we
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
53 would like to guarantee that if a transaction is committed to the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
54 database, then the edit has in fact taken place.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
55
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
56 This class provides an approximation of these transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
57 requirements.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
58
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
59 For classes that do not have a 'filename' property, the file name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
60 used to store the file's content is a deterministic function of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
61 the classname and nodeid for the file. The 'filename' function
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
62 computes this name. The name will contain directories and
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
63 subdirectories, but, suppose, for the purposes of what follows,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
64 that the filename is 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
65
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
66 Edit Procotol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
67 -------------
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
68
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
69 When a file is created or edited, the following protocol is used:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
70
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
71 1. The new content of the file is placed in 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
72
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
73 2. A transaction is recored in 'self.transactions' referencing the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
74 'doStoreFile' method of this class.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
75
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
76 3. At some subsequent point, the database 'commit' function is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
77 called. This function first performs a traditional database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
78 commit (for example, by issuing a SQL command to commit the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
79 current transaction), and, then, runs the transactions recored
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
80 in 'self.transactions'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
81
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
82 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
83
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
84 If Step 3 never occurs, but, instead, the database 'rollback'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
85 method is called, then that method, after rolling back the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
86 database transaction, calls 'rollbackStoreFile', which removes
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
87 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
88
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
89 Race Condition
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
90 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
91
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
92 If two Roundup instances (say, the mail gateway and a web client,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
93 or two web clients running with a multi-process server) attempt
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
94 edits at the same time, both will write to 'file.tmp', and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
95 results will be indeterminate.
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
96
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
97 Crash Analysis
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
98 --------------
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
99
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
100 There are several situations that may occur if a crash (whether
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
101 because the machine crashes, because an unhandled Python exception
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
102 is raised, or because the Python process is killed) occurs.
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
103
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
104 Complexity ensues because backuping up an RDBMS is generally more
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
105 complex than simply copying a file. Instead, some command is run
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
106 which stores a snapshot of the database in a file. So, if you
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
107 back up the database to a file, and then back up the filesystem,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
108 it is likely that further database transactions have occurred
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
109 between the point of database backup and the point of filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
110 backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
111
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
112 For the purposes, of this analysis, we assume that the filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
113 backup occurred after the database backup. Furthermore, we assume
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
114 that filesystem backups are atomic; i.e., the at the filesystem is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
115 not being modified during the backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
116
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
117 1. Neither the 'commit' nor 'rollback' methods on the database are
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
118 ever called.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
119
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
120 In this case, the '.tmp' file should be ignored as the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
121 transaction was not committed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
122
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
123 2. The 'commit' method is called. Subsequently, the machine
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
124 crashes, and is restored from backups.
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
125
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
126 The most recent filesystem backup and the most recent database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
127 backup are not in general from the same instant in time.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
128
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
129 This problem means that we can never be sure after a crash if
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
130 the contents of a file are what we intend. It is always
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
131 possible that an edit was made to the file that is not
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
132 reflected in the filesystem.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
133
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
134 3. A crash occurs between the point of the database commit and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
135 call to 'doStoreFile'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
136
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
137 If only one of 'file' and 'file.tmp' exists, then that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
138 version should be used. However, if both 'file' and 'file.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
139 exist, there is no way to know which version to use.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
140
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
141 Reading the File
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
142 ----------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
143
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
144 When determining the content of the file, we use the following
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
145 algorithm:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
146
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
147 1. If 'self.transactions' reflects an edit of the file, then use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
148 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
149
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
150 We know that an edit to the file is in process so 'file.tmp' is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
151 the right choice. If 'file.tmp' does not exist, raise an
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
152 exception; something has removed the content of the file while
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
153 we are in the process of editing it.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
154
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
155 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
156 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
157
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
158 We know that the file is supposed to exist because there is a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
159 reference to it in the database. Since 'file' does not exist,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
160 we assume that Crash 3 occurred during the initial creation of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
161 the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
162
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
163 3. Otherwise, use 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
164
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
165 If 'file.tmp' is not present, this is obviously the best we can
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
166 do. This is always the right answer unless Crash 2 occurred,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
167 in which case the contents of 'file' may be newer than they
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
168 were at the point of database backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
169
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
170 If 'file.tmp' is present, we know that we are not actively
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
171 editing the file. The possibilities are:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
172
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
173 a. Crash 1 has occurred. In this case, using 'file' is the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
174 right answer, so we will have chosen correctly.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
175
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
176 b. Crash 3 has occurred. In this case, 'file.tmp' is the right
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
177 answer, so we will have chosen incorrectly. However, 'file'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
178 was at least a previously committed value.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
179
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
180 Future Improvements
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
181 -------------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
182
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
183 One approach would be to take advantage of databases which do
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
184 allow the storage of arbitary date. For example, MySQL provides
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
185 the HUGE BLOB datatype for storing up to 4GB of data.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
186
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
187 Another approach would be to store a version ('v') in the actual
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
188 database and name files 'file.v'. Then, the editing protocol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
189 would become:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
190
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
191 1. Generate a new version 'v', guaranteed to be different from all
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
192 other versions ever used by the database. (The version need
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
193 not be in any particular sequence; a UUID would be fine.)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
194
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
195 2. Store the content in 'file.v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
196
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
197 3. Update the database to indicate that the version of the node is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
198 'v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
199
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
200 Now, if the transaction is committed, the database will refer to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
201 'file.v', where the content exists. If the transaction is rolled
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
202 back, or not committed, 'file.v' will never be referenced. In the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
203 event of a crash, under the assumptions above, there may be
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
204 'file.v' files that are not referenced by the database, but the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
205 database will be consistent, so long as unreferenced 'file.v'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
206 files are never removed until after the database has been backed
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
207 up.
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
208 """
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
209
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
210 tempext = '.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
211 """The suffix added to files indicating that they are uncommitted."""
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
212
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
213 def __init__(self, umask):
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
214 self.umask = umask
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
215
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
216 def subdirFilename(self, classname, nodeid, property=None):
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
217 """Determine what the filename and subdir for nodeid + classname is."""
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
218 if property:
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
219 name = '%s%s.%s' % (classname, nodeid, property)
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
220 else:
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
221 # roundupdb.FileClass never specified the property name, so don't
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
222 # include it
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
223 name = '%s%s' % (classname, nodeid)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
224
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
225 # have a separate subdir for every thousand messages
5400
2120f77554d5 Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4347
diff changeset
226 subdir = str(int(nodeid) // 1000)
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
227 return os.path.join(subdir, name)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
228
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
229 def _tempfile(self, filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
230 """Return a temporary filename.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
231
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
232 'filename' -- The name of the eventual destination file."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
233
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
234 return filename + self.tempext
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
235
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
236 def _editInProgress(self, classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
237 """Return true if the file indicated is being edited.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
238
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
239 returns -- True if the current transaction includes an edit to
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
240 the file indicated."""
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
241
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
242 for method, args in self.transactions:
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
243 if (method == self.doStoreFile and
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
244 args == (classname, nodeid, property)):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
245 return True
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
246
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
247 return False
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
248
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
249 def filename(self, classname, nodeid, property=None, create=0):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
250 """Determine what the filename for the given node and optionally
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
251 property is.
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
252
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
253 Try a variety of different filenames - the file could be in the
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
254 usual place, or it could be in a temp file pre-commit *or* it
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
255 could be in an old-style, backwards-compatible flat directory.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
256 """
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
257 filename = os.path.join(self.dir, 'files', classname,
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
258 self.subdirFilename(classname, nodeid, property))
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
259 # If the caller is going to create the file, return the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
260 # post-commit filename. It is the callers responsibility to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
261 # add self.tempext when actually creating the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
262 if create:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
263 return filename
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
264
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
265 tempfile = self._tempfile(filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
266
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
267 # If an edit to this file is in progress, then return the name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
268 # of the temporary file containing the edited content.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
269 if self._editInProgress(classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
270 if not os.path.exists(tempfile):
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
271 raise IOError('content file for %s not found' % tempfile)
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
272 return tempfile
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
273
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
274 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
275 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
276
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
277 # Otherwise, if the temporary file exists, then the probable
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
278 # explanation is that a crash occurred between the point that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
279 # the database entry recording the creation of the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
280 # occured and the point at which the file was renamed from the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
281 # temporary name to the final name.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
282 if os.path.exists(tempfile):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
283 try:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
284 # Clean up, by performing the commit now.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
285 os.rename(tempfile, filename)
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5492
diff changeset
286 except OSError:
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
287 pass
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
288 # If two Roundup clients both try to rename the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
289 # at the same time, only one of them will succeed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
290 # So, tolerate such an error -- but no other.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
291 if not os.path.exists(filename):
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
292 raise IOError('content file for %s not found' % filename)
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
293 return filename
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
294
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
295 # ok, try flat (very old-style)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
296 if property:
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
297 filename = os.path.join(self.dir, 'files', '%s%s.%s' % (
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
298 classname, nodeid, property))
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
299 else:
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
300 filename = os.path.join(self.dir, 'files', '%s%s' % (classname,
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
301 nodeid))
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
302 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
303 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
304
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
305 # file just ain't there
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
306 raise IOError('content file for %s not found' % filename)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
307
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
308 def filesize(self, classname, nodeid, property=None, create=0):
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
309 filename = self.filename(classname, nodeid, property, create)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
310 return os.path.getsize(filename)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
311
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
312 def storefile(self, classname, nodeid, property, content):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
313 """Store the content of the file in the database. The property may be
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
314 None, in which case the filename does not indicate which property
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
315 is being saved.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
316 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
317 # determine the name of the file to write to
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
318 name = self.filename(classname, nodeid, property, create=1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
319
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
320 # make sure the file storage dir exists
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
321 if not os.path.exists(os.path.dirname(name)):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
322 os.makedirs(os.path.dirname(name))
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
323
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
324 # save to a temp file
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
325 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
326
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
327 # make sure we don't register the rename action more than once
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
328 if not self._editInProgress(classname, nodeid, property):
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
329 # save off the rename action
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
330 self.transactions.append((self.doStoreFile, (classname, nodeid,
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
331 property)))
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
332 # always set umask before writing to make sure we have the proper one
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
333 # in multi-tracker (i.e. multi-umask) or modpython scenarios
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
334 # the umask may have changed since last we set it.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
335 os.umask(self.umask)
6486
8c371af7e785 Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents: 6148
diff changeset
336 fd = open(name, 'wb')
8c371af7e785 Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents: 6148
diff changeset
337 fd.write(content)
8c371af7e785 Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents: 6148
diff changeset
338 fd.close()
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
339
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
340 def getfile(self, classname, nodeid, property):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
341 """Get the content of the file in the database.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
342 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
343 filename = self.filename(classname, nodeid, property)
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
344
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
345 f = open(filename, 'rb')
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
346 try:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
347 # snarf the contents and make sure we close the file
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
348 return f.read()
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
349 finally:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
350 f.close()
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
351
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
352 def numfiles(self):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
353 """Get number of files in storage, even across subdirectories.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
354 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
355 files_dir = os.path.join(self.dir, 'files')
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
356 return files_in_dir(files_dir)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
357
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
358 def doStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
359 """Store the file as part of a transaction commit.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
360 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
361 # determine the name of the file to write to
3935
1dab48842cbd Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents: 3906
diff changeset
362 name = self.filename(classname, nodeid, property, 1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
363
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
364 # the file is currently ".tmp" - move it to its real name to commit
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
365 if name.endswith(self.tempext):
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
366 # creation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
367 dstname = os.path.splitext(name)[0]
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
368 else:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
369 # edit operation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
370 dstname = name
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
371 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
372
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
373 # content is being updated (and some platforms, eg. win32, won't
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
374 # let us rename over the top of the old file)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
375 if os.path.exists(dstname):
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
376 os.remove(dstname)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
377
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
378 os.rename(name, dstname)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
379
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
380 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
381 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
382
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
383 def rollbackStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
384 """Remove the temp file as a part of a rollback
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
385 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
386 # determine the name of the file to delete
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
387 name = self.filename(classname, nodeid, property)
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
388 if not name.endswith(self.tempext):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
389 name += self.tempext
2966
2ddba486546a various fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2962
diff changeset
390 os.remove(name)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
391
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
392 def isStoreFile(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
393 """See if there is actually any FileStorage for this node.
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
394 Is there a better way than using self.filename?
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
395 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
396 try:
6936
16d6d1f9038a flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6486
diff changeset
397 fname = self.filename(classname, nodeid) # noqa: F841
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
398 return True
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
399 except IOError:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
400 return False
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
401
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
402 def destroy(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
403 """If there is actually FileStorage for this node
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
404 remove it from the filesystem
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
405 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
406 if self.isStoreFile(classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
407 os.remove(self.filename(classname, nodeid))
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
408
652
66b324f895d1 add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 650
diff changeset
409 # vim: set filetype=python ts=4 sw=4 et si
Roundup Issue Tracker: http://roundup-tracker.org/