Mercurial > p > roundup > code

annotate roundup/backends/blobfiles.py @ 3960:bc412bb2ccd3

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Workaround race condition in file storage during transaction commit [SF#883580]
author Richard Jones <richard@users.sourceforge.net>
date Thu, 07 Feb 2008 00:57:59 +0000
parents 1dab48842cbd
children 35d2976cb4ba
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
1 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
6 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
11 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
17 #
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
18 #$Id: blobfiles.py,v 1.24 2008-02-07 00:57:59 richard Exp $
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1946
diff changeset
19 '''This module exports file storage for roundup backends.
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
20 Files are stored into a directory hierarchy.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
21 '''
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1946
diff changeset
22 __docformat__ = 'restructuredtext'
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
23
818
254b8d112eec cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
24 import os
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
25
2496
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
26 def files_in_dir(dir):
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
27 if not os.path.exists(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
28 return 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
29 num_files = 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
30 for dir_entry in os.listdir(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
31 full_filename = os.path.join(dir,dir_entry)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
32 if os.path.isfile(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
33 num_files = num_files + 1
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
34 elif os.path.isdir(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
35 num_files = num_files + files_in_dir(full_filename)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
36 return num_files
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
37
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
38 class FileStorage:
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
39 """Store files in some directory structure
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
40
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
41 Some databases do not permit the storage of arbitrary data (i.e.,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
42 file content). And, some database schema explicitly store file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
43 content in the fielsystem. In particular, if a class defines a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
44 'filename' property, it is assumed that the data is stored in the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
45 indicated file, outside of whatever database Roundup is otherwise
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
46 using.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
47
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
48 In these situations, it is difficult to maintain the transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
49 abstractions used elsewhere in Roundup. In particular, if a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
50 file's content is edited, but then the containing transaction is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
51 not committed, we do not want to commit the edit. Similarly, we
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
52 would like to guarantee that if a transaction is committed to the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
53 database, then the edit has in fact taken place.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
54
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
55 This class provides an approximation of these transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
56 requirements.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
57
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
58 For classes that do not have a 'filename' property, the file name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
59 used to store the file's content is a deterministic function of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
60 the classname and nodeid for the file. The 'filename' function
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
61 computes this name. The name will contain directories and
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
62 subdirectories, but, suppose, for the purposes of what follows,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
63 that the filename is 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
64
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
65 Edit Procotol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
66 -------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
67
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
68 When a file is created or edited, the following protocol is used:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
69
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
70 1. The new content of the file is placed in 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
71
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
72 2. A transaction is recored in 'self.transactions' referencing the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
73 'doStoreFile' method of this class.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
74
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
75 3. At some subsequent point, the database 'commit' function is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
76 called. This function first performs a traditional database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
77 commit (for example, by issuing a SQL command to commit the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
78 current transaction), and, then, runs the transactions recored
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
79 in 'self.transactions'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
80
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
81 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
82
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
83 If Step 3 never occurs, but, instead, the database 'rollback'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
84 method is called, then that method, after rolling back the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
85 database transaction, calls 'rollbackStoreFile', which removes
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
86 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
87
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
88 Race Condition
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
89 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
90
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
91 If two Roundup instances (say, the mail gateway and a web client,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
92 or two web clients running with a multi-process server) attempt
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
93 edits at the same time, both will write to 'file.tmp', and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
94 results will be indeterminate.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
95
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
96 Crash Analysis
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
97 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
98
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
99 There are several situations that may occur if a crash (whether
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
100 because the machine crashes, because an unhandled Python exception
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
101 is raised, or because the Python process is killed) occurs.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
102
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
103 Complexity ensues because backuping up an RDBMS is generally more
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
104 complex than simply copying a file. Instead, some command is run
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
105 which stores a snapshot of the database in a file. So, if you
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
106 back up the database to a file, and then back up the filesystem,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
107 it is likely that further database transactions have occurred
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
108 between the point of database backup and the point of filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
109 backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
110
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
111 For the purposes, of this analysis, we assume that the filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
112 backup occurred after the database backup. Furthermore, we assume
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
113 that filesystem backups are atomic; i.e., the at the filesystem is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
114 not being modified during the backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
115
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
116 1. Neither the 'commit' nor 'rollback' methods on the database are
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
117 ever called.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
118
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
119 In this case, the '.tmp' file should be ignored as the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
120 transaction was not committed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
121
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
122 2. The 'commit' method is called. Subsequently, the machine
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
123 crashes, and is restored from backups.
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
124
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
125 The most recent filesystem backup and the most recent database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
126 backup are not in general from the same instant in time.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
127
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
128 This problem means that we can never be sure after a crash if
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
129 the contents of a file are what we intend. It is always
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
130 possible that an edit was made to the file that is not
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
131 reflected in the filesystem.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
132
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
133 3. A crash occurs between the point of the database commit and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
134 call to 'doStoreFile'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
135
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
136 If only one of 'file' and 'file.tmp' exists, then that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
137 version should be used. However, if both 'file' and 'file.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
138 exist, there is no way to know which version to use.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
139
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
140 Reading the File
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
141 ----------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
142
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
143 When determining the content of the file, we use the following
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
144 algorithm:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
145
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
146 1. If 'self.transactions' reflects an edit of the file, then use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
147 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
148
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
149 We know that an edit to the file is in process so 'file.tmp' is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
150 the right choice. If 'file.tmp' does not exist, raise an
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
151 exception; something has removed the content of the file while
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
152 we are in the process of editing it.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
153
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
154 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
155 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
156
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
157 We know that the file is supposed to exist because there is a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
158 reference to it in the database. Since 'file' does not exist,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
159 we assume that Crash 3 occurred during the initial creation of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
160 the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
161
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
162 3. Otherwise, use 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
163
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
164 If 'file.tmp' is not present, this is obviously the best we can
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
165 do. This is always the right answer unless Crash 2 occurred,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
166 in which case the contents of 'file' may be newer than they
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
167 were at the point of database backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
168
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
169 If 'file.tmp' is present, we know that we are not actively
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
170 editing the file. The possibilities are:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
171
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
172 a. Crash 1 has occurred. In this case, using 'file' is the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
173 right answer, so we will have chosen correctly.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
174
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
175 b. Crash 3 has occurred. In this case, 'file.tmp' is the right
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
176 answer, so we will have chosen incorrectly. However, 'file'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
177 was at least a previously committed value.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
178
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
179 Future Improvements
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
180 -------------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
181
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
182 One approach would be to take advantage of databases which do
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
183 allow the storage of arbitary date. For example, MySQL provides
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
184 the HUGE BLOB datatype for storing up to 4GB of data.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
185
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
186 Another approach would be to store a version ('v') in the actual
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
187 database and name files 'file.v'. Then, the editing protocol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
188 would become:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
189
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
190 1. Generate a new version 'v', guaranteed to be different from all
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
191 other versions ever used by the database. (The version need
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
192 not be in any particular sequence; a UUID would be fine.)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
193
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
194 2. Store the content in 'file.v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
195
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
196 3. Update the database to indicate that the version of the node is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
197 'v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
198
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
199 Now, if the transaction is committed, the database will refer to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
200 'file.v', where the content exists. If the transaction is rolled
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
201 back, or not committed, 'file.v' will never be referenced. In the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
202 event of a crash, under the assumptions above, there may be
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
203 'file.v' files that are not referenced by the database, but the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
204 database will be consistent, so long as unreferenced 'file.v'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
205 files are never removed until after the database has been backed
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
206 up.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
207 """
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
208
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
209 tempext = '.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
210 """The suffix added to files indicating that they are uncommitted."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
211
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
212 def __init__(self, umask):
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
213 self.umask = umask
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
214
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
215 def subdirFilename(self, classname, nodeid, property=None):
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
216 """Determine what the filename and subdir for nodeid + classname is."""
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
217 if property:
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
218 name = '%s%s.%s'%(classname, nodeid, property)
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
219 else:
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
220 # roundupdb.FileClass never specified the property name, so don't
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
221 # include it
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
222 name = '%s%s'%(classname, nodeid)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
223
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
224 # have a separate subdir for every thousand messages
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
225 subdir = str(int(nodeid) / 1000)
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
226 return os.path.join(subdir, name)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
227
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
228 def _tempfile(self, filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
229 """Return a temporary filename.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
230
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
231 'filename' -- The name of the eventual destination file."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
232
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
233 return filename + self.tempext
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
234
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
235 def filename(self, classname, nodeid, property=None, create=0):
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
236 '''Determine what the filename for the given node and optionally
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
237 property is.
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
238
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
239 Try a variety of different filenames - the file could be in the
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
240 usual place, or it could be in a temp file pre-commit *or* it
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
241 could be in an old-style, backwards-compatible flat directory.
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
242 '''
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
243 filename = os.path.join(self.dir, 'files', classname,
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
244 self.subdirFilename(classname, nodeid, property))
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
245 # If the caller is going to create the file, return the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
246 # post-commit filename. It is the callers responsibility to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
247 # add self.tempext when actually creating the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
248 if create:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
249 return filename
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
250
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
251 tempfile = self._tempfile(filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
252
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
253 # If an edit to this file is in progress, then return the name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
254 # of the temporary file containing the edited content.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
255 for method, args in self.transactions:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
256 if (method == self.doStoreFile and
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
257 args == (classname, nodeid, property)):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
258 # There is an edit in progress for this file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
259 if not os.path.exists(tempfile):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
260 raise IOError('content file for %s not found'%tempfile)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
261 return tempfile
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
262
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
263 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
264 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
265
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
266 # Otherwise, if the temporary file exists, then the probable
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
267 # explanation is that a crash occurred between the point that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
268 # the database entry recording the creation of the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
269 # occured and the point at which the file was renamed from the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
270 # temporary name to the final name.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
271 if os.path.exists(tempfile):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
272 try:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
273 # Clean up, by performing the commit now.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
274 os.rename(tempfile, filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
275 except:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
276 pass
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
277 # If two Roundup clients both try to rename the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
278 # at the same time, only one of them will succeed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
279 # So, tolerate such an error -- but no other.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
280 if not os.path.exists(filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
281 raise IOError('content file for %s not found'%filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
282 return filename
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
283
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
284 # ok, try flat (very old-style)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
285 if property:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
286 filename = os.path.join(self.dir, 'files', '%s%s.%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
287 nodeid, property))
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
288 else:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
289 filename = os.path.join(self.dir, 'files', '%s%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
290 nodeid))
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
291 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
292 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
293
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
294 # file just ain't there
3344
2551a6f4cb78 another name error, another backport candidate
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 3019
diff changeset
295 raise IOError('content file for %s not found'%filename)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
296
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
297 def storefile(self, classname, nodeid, property, content):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
298 '''Store the content of the file in the database. The property may be
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
299 None, in which case the filename does not indicate which property
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
300 is being saved.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
301 '''
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
302 # determine the name of the file to write to
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
303 name = self.filename(classname, nodeid, property, create=1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
304
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
305 # make sure the file storage dir exists
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
306 if not os.path.exists(os.path.dirname(name)):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
307 os.makedirs(os.path.dirname(name))
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
308
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
309 # save to a temp file
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
310 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
311
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
312 # make sure we don't register the rename action more than once
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
313 if not os.path.exists(name):
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
314 # save off the rename action
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
315 self.transactions.append((self.doStoreFile, (classname, nodeid,
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
316 property)))
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
317 # always set umask before writing to make sure we have the proper one
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
318 # in multi-tracker (i.e. multi-umask) or modpython scenarios
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
319 # the umask may have changed since last we set it.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
320 os.umask(self.umask)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
321 open(name, 'wb').write(content)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
322
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
323 def getfile(self, classname, nodeid, property):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
324 '''Get the content of the file in the database.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
325 '''
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
326 filename = self.filename(classname, nodeid, property)
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
327
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
328 f = open(filename, 'rb')
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
329 try:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
330 # snarf the contents and make sure we close the file
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
331 return f.read()
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
332 finally:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
333 f.close()
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
334
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
335 def numfiles(self):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
336 '''Get number of files in storage, even across subdirectories.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
337 '''
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
338 files_dir = os.path.join(self.dir, 'files')
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
339 return files_in_dir(files_dir)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
340
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
341 def doStoreFile(self, classname, nodeid, property, **databases):
778
fb2ce2b8b2ef Moved the file storage commit into blobfiles where it belongs.
Richard Jones <richard@users.sourceforge.net>
parents: 652
diff changeset
342 '''Store the file as part of a transaction commit.
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
343 '''
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
344 # determine the name of the file to write to
3935
1dab48842cbd Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents: 3906
diff changeset
345 name = self.filename(classname, nodeid, property, 1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
346
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
347 # the file is currently ".tmp" - move it to its real name to commit
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
348 if name.endswith(self.tempext):
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
349 # creation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
350 dstname = os.path.splitext(name)[0]
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
351 else:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
352 # edit operation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
353 dstname = name
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
354 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
355
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
356 # content is being updated (and some platforms, eg. win32, won't
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
357 # let us rename over the top of the old file)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
358 if os.path.exists(dstname):
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
359 os.remove(dstname)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
360
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
361 os.rename(name, dstname)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
362
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
363 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
364 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
365
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
366 def rollbackStoreFile(self, classname, nodeid, property, **databases):
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
367 '''Remove the temp file as a part of a rollback
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
368 '''
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
369 # determine the name of the file to delete
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
370 name = self.filename(classname, nodeid, property)
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
371 if not name.endswith(self.tempext):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
372 name += self.tempext
2966
2ddba486546a various fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2962
diff changeset
373 os.remove(name)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
374
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
375 def isStoreFile(self, classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
376 '''See if there is actually any FileStorage for this node.
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
377 Is there a better way than using self.filename?
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
378 '''
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
379 try:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
380 fname = self.filename(classname, nodeid)
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
381 return True
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
382 except IOError:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
383 return False
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
384
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
385 def destroy(self, classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
386 '''If there is actually FileStorage for this node
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
387 remove it from the filesystem
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
388 '''
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
389 if self.isStoreFile(classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
390 os.remove(self.filename(classname, nodeid))
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
391
652
66b324f895d1 add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 650
diff changeset
392 # vim: set filetype=python ts=4 sw=4 et si
Roundup Issue Tracker: http://roundup-tracker.org/