Mercurial > p > roundup > code
annotate roundup/backends/blobfiles.py @ 8543:1ffa1f42e1da
refactor: rework mime type comparison and clean code
rest.py:
accept application/* as match for application/json in non
/binary_context rest path.
allow defining default mime type to return when file/message is
missing mime type. Make it a class variable to it can be changed from
text/plain to text/markdown or whatever.
extract code from determine_output_format() to create
create_valid_content_types() method which returns a list of matching
mime types for a given type/subtype.
Eliminate mostly duplicate return statements by introducing a variable
to specify valid mime types in error message.
rest_common.py:
Fix error messages that now return application/* as valid mime type.
CHANGES.txt upgrading.txt rest.txt:
top level notes and corrections.
Also correct rst syntax on earlier change.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Tue, 24 Mar 2026 21:30:47 -0400 |
| parents | 586f76eb33e8 |
| children |
| rev | line source |
|---|---|
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
1 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
6 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
11 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
17 # |
| 4084 | 18 """This module exports file storage for roundup backends. |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
19 Files are stored into a directory hierarchy. |
| 4084 | 20 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1946
diff
changeset
|
21 __docformat__ = 'restructuredtext' |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
22 |
|
818
254b8d112eec
cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents:
778
diff
changeset
|
23 import os |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
24 |
|
8091
586f76eb33e8
fix: keep python2 working a little longer.
John Rouillard <rouilj@ieee.org>
parents:
8088
diff
changeset
|
25 from roundup.anypy import scandir_ |
| 6936 | 26 |
| 2496 | 27 def files_in_dir(dir): |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
28 if not os.path.exists(dir): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
29 return 0 |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
30 num_files = 0 |
|
8088
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
31 for dir_entry in os.scandir(dir): |
|
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
32 if dir_entry.is_file(): |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
33 num_files = num_files + 1 |
|
8088
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
34 elif dir_entry.is_dir(): |
|
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
35 num_files = num_files + files_in_dir(dir_entry.path) |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
36 return num_files |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
37 |
| 6936 | 38 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6002
diff
changeset
|
39 class FileStorage(object): |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
40 """Store files in some directory structure |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
41 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
42 Some databases do not permit the storage of arbitrary data (i.e., |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
43 file content). And, some database schema explicitly store file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
44 content in the fielsystem. In particular, if a class defines a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
45 'filename' property, it is assumed that the data is stored in the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
46 indicated file, outside of whatever database Roundup is otherwise |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
47 using. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
48 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
49 In these situations, it is difficult to maintain the transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
50 abstractions used elsewhere in Roundup. In particular, if a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
51 file's content is edited, but then the containing transaction is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
52 not committed, we do not want to commit the edit. Similarly, we |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
53 would like to guarantee that if a transaction is committed to the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
54 database, then the edit has in fact taken place. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
55 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
56 This class provides an approximation of these transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
57 requirements. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
58 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
59 For classes that do not have a 'filename' property, the file name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
60 used to store the file's content is a deterministic function of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
61 the classname and nodeid for the file. The 'filename' function |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
62 computes this name. The name will contain directories and |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
63 subdirectories, but, suppose, for the purposes of what follows, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
64 that the filename is 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
65 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
66 Edit Procotol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
67 ------------- |
| 6936 | 68 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
69 When a file is created or edited, the following protocol is used: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
70 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
71 1. The new content of the file is placed in 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
72 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
73 2. A transaction is recored in 'self.transactions' referencing the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
74 'doStoreFile' method of this class. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
75 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
76 3. At some subsequent point, the database 'commit' function is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
77 called. This function first performs a traditional database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
78 commit (for example, by issuing a SQL command to commit the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
79 current transaction), and, then, runs the transactions recored |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
80 in 'self.transactions'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
81 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
82 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
83 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
84 If Step 3 never occurs, but, instead, the database 'rollback' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
85 method is called, then that method, after rolling back the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
86 database transaction, calls 'rollbackStoreFile', which removes |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
87 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
88 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
89 Race Condition |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
90 -------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
91 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
92 If two Roundup instances (say, the mail gateway and a web client, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
93 or two web clients running with a multi-process server) attempt |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
94 edits at the same time, both will write to 'file.tmp', and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
95 results will be indeterminate. |
| 6936 | 96 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
97 Crash Analysis |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
98 -------------- |
| 6936 | 99 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
100 There are several situations that may occur if a crash (whether |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
101 because the machine crashes, because an unhandled Python exception |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
102 is raised, or because the Python process is killed) occurs. |
| 6936 | 103 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
104 Complexity ensues because backuping up an RDBMS is generally more |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
105 complex than simply copying a file. Instead, some command is run |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
106 which stores a snapshot of the database in a file. So, if you |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
107 back up the database to a file, and then back up the filesystem, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
108 it is likely that further database transactions have occurred |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
109 between the point of database backup and the point of filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
110 backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
111 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
112 For the purposes, of this analysis, we assume that the filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
113 backup occurred after the database backup. Furthermore, we assume |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
114 that filesystem backups are atomic; i.e., the at the filesystem is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
115 not being modified during the backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
116 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
117 1. Neither the 'commit' nor 'rollback' methods on the database are |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
118 ever called. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
119 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
120 In this case, the '.tmp' file should be ignored as the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
121 transaction was not committed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
122 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
123 2. The 'commit' method is called. Subsequently, the machine |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
124 crashes, and is restored from backups. |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
125 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
126 The most recent filesystem backup and the most recent database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
127 backup are not in general from the same instant in time. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
128 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
129 This problem means that we can never be sure after a crash if |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
130 the contents of a file are what we intend. It is always |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
131 possible that an edit was made to the file that is not |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
132 reflected in the filesystem. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
133 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
134 3. A crash occurs between the point of the database commit and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
135 call to 'doStoreFile'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
136 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
137 If only one of 'file' and 'file.tmp' exists, then that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
138 version should be used. However, if both 'file' and 'file.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
139 exist, there is no way to know which version to use. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
140 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
141 Reading the File |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
142 ---------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
143 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
144 When determining the content of the file, we use the following |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
145 algorithm: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
146 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
147 1. If 'self.transactions' reflects an edit of the file, then use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
148 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
149 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
150 We know that an edit to the file is in process so 'file.tmp' is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
151 the right choice. If 'file.tmp' does not exist, raise an |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
152 exception; something has removed the content of the file while |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
153 we are in the process of editing it. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
154 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
155 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
156 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
157 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
158 We know that the file is supposed to exist because there is a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
159 reference to it in the database. Since 'file' does not exist, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
160 we assume that Crash 3 occurred during the initial creation of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
161 the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
162 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
163 3. Otherwise, use 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
164 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
165 If 'file.tmp' is not present, this is obviously the best we can |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
166 do. This is always the right answer unless Crash 2 occurred, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
167 in which case the contents of 'file' may be newer than they |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
168 were at the point of database backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
169 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
170 If 'file.tmp' is present, we know that we are not actively |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
171 editing the file. The possibilities are: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
172 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
173 a. Crash 1 has occurred. In this case, using 'file' is the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
174 right answer, so we will have chosen correctly. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
175 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
176 b. Crash 3 has occurred. In this case, 'file.tmp' is the right |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
177 answer, so we will have chosen incorrectly. However, 'file' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
178 was at least a previously committed value. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
179 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
180 Future Improvements |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
181 ------------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
182 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
183 One approach would be to take advantage of databases which do |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
184 allow the storage of arbitary date. For example, MySQL provides |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
185 the HUGE BLOB datatype for storing up to 4GB of data. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
186 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
187 Another approach would be to store a version ('v') in the actual |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
188 database and name files 'file.v'. Then, the editing protocol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
189 would become: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
190 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
191 1. Generate a new version 'v', guaranteed to be different from all |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
192 other versions ever used by the database. (The version need |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
193 not be in any particular sequence; a UUID would be fine.) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
194 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
195 2. Store the content in 'file.v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
196 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
197 3. Update the database to indicate that the version of the node is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
198 'v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
199 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
200 Now, if the transaction is committed, the database will refer to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
201 'file.v', where the content exists. If the transaction is rolled |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
202 back, or not committed, 'file.v' will never be referenced. In the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
203 event of a crash, under the assumptions above, there may be |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
204 'file.v' files that are not referenced by the database, but the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
205 database will be consistent, so long as unreferenced 'file.v' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
206 files are never removed until after the database has been backed |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
207 up. |
| 6936 | 208 """ |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
209 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
210 tempext = '.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
211 """The suffix added to files indicating that they are uncommitted.""" |
| 6936 | 212 |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
213 def __init__(self, umask): |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
214 self.umask = umask |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
215 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
216 def subdirFilename(self, classname, nodeid, property=None): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
217 """Determine what the filename and subdir for nodeid + classname is.""" |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
218 if property: |
| 6936 | 219 name = '%s%s.%s' % (classname, nodeid, property) |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
220 else: |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
221 # roundupdb.FileClass never specified the property name, so don't |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
222 # include it |
| 6936 | 223 name = '%s%s' % (classname, nodeid) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
224 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
225 # have a separate subdir for every thousand messages |
|
5400
2120f77554d5
Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4347
diff
changeset
|
226 subdir = str(int(nodeid) // 1000) |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
227 return os.path.join(subdir, name) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
228 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
229 def _tempfile(self, filename): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
230 """Return a temporary filename. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
231 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
232 'filename' -- The name of the eventual destination file.""" |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
233 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
234 return filename + self.tempext |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
235 |
| 4084 | 236 def _editInProgress(self, classname, nodeid, property): |
| 237 """Return true if the file indicated is being edited. | |
| 238 | |
| 239 returns -- True if the current transaction includes an edit to | |
| 240 the file indicated.""" | |
| 241 | |
| 242 for method, args in self.transactions: | |
| 243 if (method == self.doStoreFile and | |
| 6936 | 244 args == (classname, nodeid, property)): |
| 4084 | 245 return True |
| 246 | |
| 247 return False | |
| 248 | |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
249 def filename(self, classname, nodeid, property=None, create=0): |
| 4084 | 250 """Determine what the filename for the given node and optionally |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
251 property is. |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
252 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
253 Try a variety of different filenames - the file could be in the |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
254 usual place, or it could be in a temp file pre-commit *or* it |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
255 could be in an old-style, backwards-compatible flat directory. |
| 4084 | 256 """ |
| 6936 | 257 filename = os.path.join(self.dir, 'files', classname, |
| 258 self.subdirFilename(classname, nodeid, property)) | |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
259 # If the caller is going to create the file, return the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
260 # post-commit filename. It is the callers responsibility to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
261 # add self.tempext when actually creating the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
262 if create: |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
263 return filename |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
264 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
265 tempfile = self._tempfile(filename) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
266 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
267 # If an edit to this file is in progress, then return the name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
268 # of the temporary file containing the edited content. |
| 4084 | 269 if self._editInProgress(classname, nodeid, property): |
| 270 if not os.path.exists(tempfile): | |
| 6936 | 271 raise IOError('content file for %s not found' % tempfile) |
| 4084 | 272 return tempfile |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
273 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
274 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
275 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
276 |
| 6936 | 277 # Otherwise, if the temporary file exists, then the probable |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
278 # explanation is that a crash occurred between the point that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
279 # the database entry recording the creation of the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
280 # occured and the point at which the file was renamed from the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
281 # temporary name to the final name. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
282 if os.path.exists(tempfile): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
283 try: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
284 # Clean up, by performing the commit now. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
285 os.rename(tempfile, filename) |
| 6002 | 286 except OSError: |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
287 pass |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
288 # If two Roundup clients both try to rename the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
289 # at the same time, only one of them will succeed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
290 # So, tolerate such an error -- but no other. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
291 if not os.path.exists(filename): |
| 6936 | 292 raise IOError('content file for %s not found' % filename) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
293 return filename |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
294 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
295 # ok, try flat (very old-style) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
296 if property: |
| 6936 | 297 filename = os.path.join(self.dir, 'files', '%s%s.%s' % ( |
| 298 classname, nodeid, property)) | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
299 else: |
| 6936 | 300 filename = os.path.join(self.dir, 'files', '%s%s' % (classname, |
| 301 nodeid)) | |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
302 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
303 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
304 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
305 # file just ain't there |
| 6936 | 306 raise IOError('content file for %s not found' % filename) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
307 |
|
4347
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
308 def filesize(self, classname, nodeid, property=None, create=0): |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
309 filename = self.filename(classname, nodeid, property, create) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
310 return os.path.getsize(filename) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
311 |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
312 def storefile(self, classname, nodeid, property, content): |
| 4084 | 313 """Store the content of the file in the database. The property may be |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
314 None, in which case the filename does not indicate which property |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
315 is being saved. |
| 4084 | 316 """ |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
317 # determine the name of the file to write to |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
318 name = self.filename(classname, nodeid, property, create=1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
319 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
320 # make sure the file storage dir exists |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
321 if not os.path.exists(os.path.dirname(name)): |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
322 os.makedirs(os.path.dirname(name)) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
323 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
324 # save to a temp file |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
325 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
326 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
327 # make sure we don't register the rename action more than once |
| 4084 | 328 if not self._editInProgress(classname, nodeid, property): |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
329 # save off the rename action |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
330 self.transactions.append((self.doStoreFile, (classname, nodeid, |
| 6936 | 331 property))) |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
332 # always set umask before writing to make sure we have the proper one |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
333 # in multi-tracker (i.e. multi-umask) or modpython scenarios |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
334 # the umask may have changed since last we set it. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
335 os.umask(self.umask) |
|
6486
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
336 fd = open(name, 'wb') |
|
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
337 fd.write(content) |
|
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
338 fd.close() |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
339 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
340 def getfile(self, classname, nodeid, property): |
| 4084 | 341 """Get the content of the file in the database. |
| 342 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
343 filename = self.filename(classname, nodeid, property) |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
344 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
345 f = open(filename, 'rb') |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
346 try: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
347 # snarf the contents and make sure we close the file |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
348 return f.read() |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
349 finally: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
350 f.close() |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
351 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
352 def numfiles(self): |
| 4084 | 353 """Get number of files in storage, even across subdirectories. |
| 354 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
355 files_dir = os.path.join(self.dir, 'files') |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
356 return files_in_dir(files_dir) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
357 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
358 def doStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 359 """Store the file as part of a transaction commit. |
| 360 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
361 # determine the name of the file to write to |
|
3935
1dab48842cbd
Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents:
3906
diff
changeset
|
362 name = self.filename(classname, nodeid, property, 1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
363 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
364 # the file is currently ".tmp" - move it to its real name to commit |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
365 if name.endswith(self.tempext): |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
366 # creation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
367 dstname = os.path.splitext(name)[0] |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
368 else: |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
369 # edit operation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
370 dstname = name |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
371 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
372 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
373 # content is being updated (and some platforms, eg. win32, won't |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
374 # let us rename over the top of the old file) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
375 if os.path.exists(dstname): |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
376 os.remove(dstname) |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
377 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
378 os.rename(name, dstname) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
379 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
380 # return the classname, nodeid so we reindex this content |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
381 return (classname, nodeid) |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
382 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
383 def rollbackStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 384 """Remove the temp file as a part of a rollback |
| 385 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
386 # determine the name of the file to delete |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
387 name = self.filename(classname, nodeid, property) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
388 if not name.endswith(self.tempext): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
389 name += self.tempext |
|
2966
2ddba486546a
various fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2962
diff
changeset
|
390 os.remove(name) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
391 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
392 def isStoreFile(self, classname, nodeid): |
| 4084 | 393 """See if there is actually any FileStorage for this node. |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
394 Is there a better way than using self.filename? |
| 4084 | 395 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
396 try: |
| 6936 | 397 fname = self.filename(classname, nodeid) # noqa: F841 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
398 return True |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
399 except IOError: |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
400 return False |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
401 |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
402 def destroy(self, classname, nodeid): |
| 4084 | 403 """If there is actually FileStorage for this node |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
404 remove it from the filesystem |
| 4084 | 405 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
406 if self.isStoreFile(classname, nodeid): |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
407 os.remove(self.filename(classname, nodeid)) |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
408 |
|
652
66b324f895d1
add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
650
diff
changeset
|
409 # vim: set filetype=python ts=4 sw=4 et si |