Mercurial > p > roundup > code

annotate roundup/backends/blobfiles.py @ 5400:2120f77554d5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Python 3 preparation: use // and __truediv__ as needed. Tool-assisted patch. Those divisions that I thought must be integer floor divisions and rely on Python 2 integer floor division semantics are changed to use // (if any are actually meant to be floating-point divisions, that would break things). One __div__ method is changed to __truediv__ (with __div__ = __truediv__ for Python 2 compatibility).
author Joseph Myers <jsm@polyomino.org.uk>
date Tue, 24 Jul 2018 23:16:09 +0000
parents 0e33bf5571dc
children 46317fe544ec
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
1 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
6 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
11 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
17 #
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
18 """This module exports file storage for roundup backends.
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
19 Files are stored into a directory hierarchy.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
20 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1946
diff changeset
21 __docformat__ = 'restructuredtext'
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
22
818
254b8d112eec cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
23 import os
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
24
2496
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
25 def files_in_dir(dir):
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
26 if not os.path.exists(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
27 return 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
28 num_files = 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
29 for dir_entry in os.listdir(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
30 full_filename = os.path.join(dir,dir_entry)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
31 if os.path.isfile(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
32 num_files = num_files + 1
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
33 elif os.path.isdir(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
34 num_files = num_files + files_in_dir(full_filename)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
35 return num_files
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
36
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
37 class FileStorage:
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
38 """Store files in some directory structure
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
39
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
40 Some databases do not permit the storage of arbitrary data (i.e.,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
41 file content). And, some database schema explicitly store file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
42 content in the fielsystem. In particular, if a class defines a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
43 'filename' property, it is assumed that the data is stored in the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
44 indicated file, outside of whatever database Roundup is otherwise
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
45 using.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
46
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
47 In these situations, it is difficult to maintain the transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
48 abstractions used elsewhere in Roundup. In particular, if a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
49 file's content is edited, but then the containing transaction is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
50 not committed, we do not want to commit the edit. Similarly, we
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
51 would like to guarantee that if a transaction is committed to the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
52 database, then the edit has in fact taken place.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
53
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
54 This class provides an approximation of these transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
55 requirements.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
56
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
57 For classes that do not have a 'filename' property, the file name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
58 used to store the file's content is a deterministic function of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
59 the classname and nodeid for the file. The 'filename' function
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
60 computes this name. The name will contain directories and
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
61 subdirectories, but, suppose, for the purposes of what follows,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
62 that the filename is 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
63
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
64 Edit Procotol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
65 -------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
66
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
67 When a file is created or edited, the following protocol is used:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
68
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
69 1. The new content of the file is placed in 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
70
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
71 2. A transaction is recored in 'self.transactions' referencing the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
72 'doStoreFile' method of this class.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
73
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
74 3. At some subsequent point, the database 'commit' function is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
75 called. This function first performs a traditional database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
76 commit (for example, by issuing a SQL command to commit the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
77 current transaction), and, then, runs the transactions recored
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
78 in 'self.transactions'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
79
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
80 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
81
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
82 If Step 3 never occurs, but, instead, the database 'rollback'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
83 method is called, then that method, after rolling back the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
84 database transaction, calls 'rollbackStoreFile', which removes
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
85 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
86
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
87 Race Condition
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
88 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
89
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
90 If two Roundup instances (say, the mail gateway and a web client,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
91 or two web clients running with a multi-process server) attempt
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
92 edits at the same time, both will write to 'file.tmp', and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
93 results will be indeterminate.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
94
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
95 Crash Analysis
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
96 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
97
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
98 There are several situations that may occur if a crash (whether
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
99 because the machine crashes, because an unhandled Python exception
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
100 is raised, or because the Python process is killed) occurs.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
101
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
102 Complexity ensues because backuping up an RDBMS is generally more
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
103 complex than simply copying a file. Instead, some command is run
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
104 which stores a snapshot of the database in a file. So, if you
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
105 back up the database to a file, and then back up the filesystem,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
106 it is likely that further database transactions have occurred
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
107 between the point of database backup and the point of filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
108 backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
109
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
110 For the purposes, of this analysis, we assume that the filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
111 backup occurred after the database backup. Furthermore, we assume
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
112 that filesystem backups are atomic; i.e., the at the filesystem is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
113 not being modified during the backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
114
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
115 1. Neither the 'commit' nor 'rollback' methods on the database are
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
116 ever called.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
117
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
118 In this case, the '.tmp' file should be ignored as the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
119 transaction was not committed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
120
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
121 2. The 'commit' method is called. Subsequently, the machine
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
122 crashes, and is restored from backups.
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
123
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
124 The most recent filesystem backup and the most recent database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
125 backup are not in general from the same instant in time.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
126
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
127 This problem means that we can never be sure after a crash if
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
128 the contents of a file are what we intend. It is always
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
129 possible that an edit was made to the file that is not
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
130 reflected in the filesystem.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
131
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
132 3. A crash occurs between the point of the database commit and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
133 call to 'doStoreFile'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
134
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
135 If only one of 'file' and 'file.tmp' exists, then that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
136 version should be used. However, if both 'file' and 'file.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
137 exist, there is no way to know which version to use.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
138
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
139 Reading the File
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
140 ----------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
141
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
142 When determining the content of the file, we use the following
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
143 algorithm:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
144
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
145 1. If 'self.transactions' reflects an edit of the file, then use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
146 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
147
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
148 We know that an edit to the file is in process so 'file.tmp' is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
149 the right choice. If 'file.tmp' does not exist, raise an
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
150 exception; something has removed the content of the file while
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
151 we are in the process of editing it.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
152
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
153 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
154 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
155
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
156 We know that the file is supposed to exist because there is a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
157 reference to it in the database. Since 'file' does not exist,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
158 we assume that Crash 3 occurred during the initial creation of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
159 the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
160
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
161 3. Otherwise, use 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
162
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
163 If 'file.tmp' is not present, this is obviously the best we can
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
164 do. This is always the right answer unless Crash 2 occurred,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
165 in which case the contents of 'file' may be newer than they
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
166 were at the point of database backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
167
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
168 If 'file.tmp' is present, we know that we are not actively
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
169 editing the file. The possibilities are:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
170
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
171 a. Crash 1 has occurred. In this case, using 'file' is the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
172 right answer, so we will have chosen correctly.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
173
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
174 b. Crash 3 has occurred. In this case, 'file.tmp' is the right
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
175 answer, so we will have chosen incorrectly. However, 'file'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
176 was at least a previously committed value.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
177
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
178 Future Improvements
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
179 -------------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
180
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
181 One approach would be to take advantage of databases which do
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
182 allow the storage of arbitary date. For example, MySQL provides
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
183 the HUGE BLOB datatype for storing up to 4GB of data.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
184
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
185 Another approach would be to store a version ('v') in the actual
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
186 database and name files 'file.v'. Then, the editing protocol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
187 would become:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
188
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
189 1. Generate a new version 'v', guaranteed to be different from all
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
190 other versions ever used by the database. (The version need
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
191 not be in any particular sequence; a UUID would be fine.)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
192
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
193 2. Store the content in 'file.v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
194
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
195 3. Update the database to indicate that the version of the node is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
196 'v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
197
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
198 Now, if the transaction is committed, the database will refer to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
199 'file.v', where the content exists. If the transaction is rolled
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
200 back, or not committed, 'file.v' will never be referenced. In the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
201 event of a crash, under the assumptions above, there may be
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
202 'file.v' files that are not referenced by the database, but the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
203 database will be consistent, so long as unreferenced 'file.v'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
204 files are never removed until after the database has been backed
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
205 up.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
206 """
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
207
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
208 tempext = '.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
209 """The suffix added to files indicating that they are uncommitted."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
210
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
211 def __init__(self, umask):
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
212 self.umask = umask
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
213
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
214 def subdirFilename(self, classname, nodeid, property=None):
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
215 """Determine what the filename and subdir for nodeid + classname is."""
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
216 if property:
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
217 name = '%s%s.%s'%(classname, nodeid, property)
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
218 else:
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
219 # roundupdb.FileClass never specified the property name, so don't
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
220 # include it
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
221 name = '%s%s'%(classname, nodeid)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
222
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
223 # have a separate subdir for every thousand messages
5400
2120f77554d5 Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4347
diff changeset
224 subdir = str(int(nodeid) // 1000)
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
225 return os.path.join(subdir, name)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
226
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
227 def _tempfile(self, filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
228 """Return a temporary filename.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
229
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
230 'filename' -- The name of the eventual destination file."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
231
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
232 return filename + self.tempext
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
233
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
234 def _editInProgress(self, classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
235 """Return true if the file indicated is being edited.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
236
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
237 returns -- True if the current transaction includes an edit to
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
238 the file indicated."""
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
239
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
240 for method, args in self.transactions:
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
241 if (method == self.doStoreFile and
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
242 args == (classname, nodeid, property)):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
243 return True
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
244
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
245 return False
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
246
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
247
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
248 def filename(self, classname, nodeid, property=None, create=0):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
249 """Determine what the filename for the given node and optionally
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
250 property is.
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
251
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
252 Try a variety of different filenames - the file could be in the
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
253 usual place, or it could be in a temp file pre-commit *or* it
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
254 could be in an old-style, backwards-compatible flat directory.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
255 """
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
256 filename = os.path.join(self.dir, 'files', classname,
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
257 self.subdirFilename(classname, nodeid, property))
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
258 # If the caller is going to create the file, return the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
259 # post-commit filename. It is the callers responsibility to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
260 # add self.tempext when actually creating the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
261 if create:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
262 return filename
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
263
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
264 tempfile = self._tempfile(filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
265
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
266 # If an edit to this file is in progress, then return the name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
267 # of the temporary file containing the edited content.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
268 if self._editInProgress(classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
269 if not os.path.exists(tempfile):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
270 raise IOError('content file for %s not found'%tempfile)
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
271 return tempfile
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
272
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
273 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
274 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
275
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
276 # Otherwise, if the temporary file exists, then the probable
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
277 # explanation is that a crash occurred between the point that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
278 # the database entry recording the creation of the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
279 # occured and the point at which the file was renamed from the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
280 # temporary name to the final name.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
281 if os.path.exists(tempfile):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
282 try:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
283 # Clean up, by performing the commit now.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
284 os.rename(tempfile, filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
285 except:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
286 pass
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
287 # If two Roundup clients both try to rename the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
288 # at the same time, only one of them will succeed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
289 # So, tolerate such an error -- but no other.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
290 if not os.path.exists(filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
291 raise IOError('content file for %s not found'%filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
292 return filename
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
293
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
294 # ok, try flat (very old-style)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
295 if property:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
296 filename = os.path.join(self.dir, 'files', '%s%s.%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
297 nodeid, property))
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
298 else:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
299 filename = os.path.join(self.dir, 'files', '%s%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
300 nodeid))
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
301 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
302 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
303
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
304 # file just ain't there
3344
2551a6f4cb78 another name error, another backport candidate
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 3019
diff changeset
305 raise IOError('content file for %s not found'%filename)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
306
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
307 def filesize(self, classname, nodeid, property=None, create=0):
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
308 filename = self.filename(classname, nodeid, property, create)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
309 return os.path.getsize(filename)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
310
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
311 def storefile(self, classname, nodeid, property, content):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
312 """Store the content of the file in the database. The property may be
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
313 None, in which case the filename does not indicate which property
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
314 is being saved.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
315 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
316 # determine the name of the file to write to
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
317 name = self.filename(classname, nodeid, property, create=1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
318
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
319 # make sure the file storage dir exists
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
320 if not os.path.exists(os.path.dirname(name)):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
321 os.makedirs(os.path.dirname(name))
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
322
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
323 # save to a temp file
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
324 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
325
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
326 # make sure we don't register the rename action more than once
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
327 if not self._editInProgress(classname, nodeid, property):
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
328 # save off the rename action
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
329 self.transactions.append((self.doStoreFile, (classname, nodeid,
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
330 property)))
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
331 # always set umask before writing to make sure we have the proper one
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
332 # in multi-tracker (i.e. multi-umask) or modpython scenarios
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
333 # the umask may have changed since last we set it.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
334 os.umask(self.umask)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
335 open(name, 'wb').write(content)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
336
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
337 def getfile(self, classname, nodeid, property):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
338 """Get the content of the file in the database.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
339 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
340 filename = self.filename(classname, nodeid, property)
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
341
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
342 f = open(filename, 'rb')
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
343 try:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
344 # snarf the contents and make sure we close the file
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
345 return f.read()
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
346 finally:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
347 f.close()
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
348
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
349 def numfiles(self):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
350 """Get number of files in storage, even across subdirectories.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
351 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
352 files_dir = os.path.join(self.dir, 'files')
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
353 return files_in_dir(files_dir)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
354
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
355 def doStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
356 """Store the file as part of a transaction commit.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
357 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
358 # determine the name of the file to write to
3935
1dab48842cbd Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents: 3906
diff changeset
359 name = self.filename(classname, nodeid, property, 1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
360
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
361 # the file is currently ".tmp" - move it to its real name to commit
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
362 if name.endswith(self.tempext):
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
363 # creation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
364 dstname = os.path.splitext(name)[0]
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
365 else:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
366 # edit operation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
367 dstname = name
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
368 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
369
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
370 # content is being updated (and some platforms, eg. win32, won't
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
371 # let us rename over the top of the old file)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
372 if os.path.exists(dstname):
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
373 os.remove(dstname)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
374
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
375 os.rename(name, dstname)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
376
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
377 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
378 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
379
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
380 def rollbackStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
381 """Remove the temp file as a part of a rollback
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
382 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
383 # determine the name of the file to delete
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
384 name = self.filename(classname, nodeid, property)
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
385 if not name.endswith(self.tempext):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
386 name += self.tempext
2966
2ddba486546a various fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2962
diff changeset
387 os.remove(name)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
388
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
389 def isStoreFile(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
390 """See if there is actually any FileStorage for this node.
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
391 Is there a better way than using self.filename?
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
392 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
393 try:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
394 fname = self.filename(classname, nodeid)
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
395 return True
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
396 except IOError:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
397 return False
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
398
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
399 def destroy(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
400 """If there is actually FileStorage for this node
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
401 remove it from the filesystem
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
402 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
403 if self.isStoreFile(classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
404 os.remove(self.filename(classname, nodeid))
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
405
652
66b324f895d1 add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 650
diff changeset
406 # vim: set filetype=python ts=4 sw=4 et si
Roundup Issue Tracker: http://roundup-tracker.org/