Mercurial > p > roundup > code
annotate roundup/backends/blobfiles.py @ 8088:1045425c23b2
refactor!: replace os.listdir() with os.scandir()
In many places we did a listdir() then a stat to see if it's a file or
directory. This change removes the need for the stat call. Also for
larger directories, scandir() is an iterator, so less memory use.
There is one remnant of listdir used in an error handler. That
requires a stat on each element in the directory, so there is no
benefit to using scandir() other than a slight memory saving on a
rarely used piece of code.
BREAKING CHANGE:
Python 2 requires installation of scandir pip package after this
commit.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Tue, 16 Jul 2024 01:05:49 -0400 |
| parents | 16d6d1f9038a |
| children | 586f76eb33e8 |
| rev | line source |
|---|---|
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
1 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
6 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
11 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
17 # |
| 4084 | 18 """This module exports file storage for roundup backends. |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
19 Files are stored into a directory hierarchy. |
| 4084 | 20 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1946
diff
changeset
|
21 __docformat__ = 'restructuredtext' |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
22 |
|
818
254b8d112eec
cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents:
778
diff
changeset
|
23 import os |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
24 |
| 6936 | 25 |
| 2496 | 26 def files_in_dir(dir): |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
27 if not os.path.exists(dir): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
28 return 0 |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
29 num_files = 0 |
|
8088
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
30 for dir_entry in os.scandir(dir): |
|
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
31 if dir_entry.is_file(): |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
32 num_files = num_files + 1 |
|
8088
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
33 elif dir_entry.is_dir(): |
|
1045425c23b2
refactor!: replace os.listdir() with os.scandir()
John Rouillard <rouilj@ieee.org>
parents:
6936
diff
changeset
|
34 num_files = num_files + files_in_dir(dir_entry.path) |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
35 return num_files |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
36 |
| 6936 | 37 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6002
diff
changeset
|
38 class FileStorage(object): |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
39 """Store files in some directory structure |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
40 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
41 Some databases do not permit the storage of arbitrary data (i.e., |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
42 file content). And, some database schema explicitly store file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
43 content in the fielsystem. In particular, if a class defines a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
44 'filename' property, it is assumed that the data is stored in the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
45 indicated file, outside of whatever database Roundup is otherwise |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
46 using. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
47 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
48 In these situations, it is difficult to maintain the transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
49 abstractions used elsewhere in Roundup. In particular, if a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
50 file's content is edited, but then the containing transaction is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
51 not committed, we do not want to commit the edit. Similarly, we |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
52 would like to guarantee that if a transaction is committed to the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
53 database, then the edit has in fact taken place. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
54 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
55 This class provides an approximation of these transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
56 requirements. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
57 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
58 For classes that do not have a 'filename' property, the file name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
59 used to store the file's content is a deterministic function of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
60 the classname and nodeid for the file. The 'filename' function |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
61 computes this name. The name will contain directories and |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
62 subdirectories, but, suppose, for the purposes of what follows, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
63 that the filename is 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
64 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
65 Edit Procotol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
66 ------------- |
| 6936 | 67 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
68 When a file is created or edited, the following protocol is used: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
69 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
70 1. The new content of the file is placed in 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
71 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
72 2. A transaction is recored in 'self.transactions' referencing the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
73 'doStoreFile' method of this class. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
74 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
75 3. At some subsequent point, the database 'commit' function is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
76 called. This function first performs a traditional database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
77 commit (for example, by issuing a SQL command to commit the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
78 current transaction), and, then, runs the transactions recored |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
79 in 'self.transactions'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
80 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
81 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
82 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
83 If Step 3 never occurs, but, instead, the database 'rollback' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
84 method is called, then that method, after rolling back the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
85 database transaction, calls 'rollbackStoreFile', which removes |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
86 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
87 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
88 Race Condition |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
89 -------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
90 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
91 If two Roundup instances (say, the mail gateway and a web client, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
92 or two web clients running with a multi-process server) attempt |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
93 edits at the same time, both will write to 'file.tmp', and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
94 results will be indeterminate. |
| 6936 | 95 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
96 Crash Analysis |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
97 -------------- |
| 6936 | 98 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
99 There are several situations that may occur if a crash (whether |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
100 because the machine crashes, because an unhandled Python exception |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
101 is raised, or because the Python process is killed) occurs. |
| 6936 | 102 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
103 Complexity ensues because backuping up an RDBMS is generally more |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
104 complex than simply copying a file. Instead, some command is run |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
105 which stores a snapshot of the database in a file. So, if you |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
106 back up the database to a file, and then back up the filesystem, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
107 it is likely that further database transactions have occurred |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
108 between the point of database backup and the point of filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
109 backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
110 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
111 For the purposes, of this analysis, we assume that the filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
112 backup occurred after the database backup. Furthermore, we assume |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
113 that filesystem backups are atomic; i.e., the at the filesystem is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
114 not being modified during the backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
115 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
116 1. Neither the 'commit' nor 'rollback' methods on the database are |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
117 ever called. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
118 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
119 In this case, the '.tmp' file should be ignored as the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
120 transaction was not committed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
121 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
122 2. The 'commit' method is called. Subsequently, the machine |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
123 crashes, and is restored from backups. |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
124 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
125 The most recent filesystem backup and the most recent database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
126 backup are not in general from the same instant in time. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
127 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
128 This problem means that we can never be sure after a crash if |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
129 the contents of a file are what we intend. It is always |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
130 possible that an edit was made to the file that is not |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
131 reflected in the filesystem. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
132 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
133 3. A crash occurs between the point of the database commit and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
134 call to 'doStoreFile'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
135 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
136 If only one of 'file' and 'file.tmp' exists, then that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
137 version should be used. However, if both 'file' and 'file.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
138 exist, there is no way to know which version to use. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
139 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
140 Reading the File |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
141 ---------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
142 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
143 When determining the content of the file, we use the following |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
144 algorithm: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
145 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
146 1. If 'self.transactions' reflects an edit of the file, then use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
147 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
148 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
149 We know that an edit to the file is in process so 'file.tmp' is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
150 the right choice. If 'file.tmp' does not exist, raise an |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
151 exception; something has removed the content of the file while |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
152 we are in the process of editing it. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
153 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
154 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
155 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
156 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
157 We know that the file is supposed to exist because there is a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
158 reference to it in the database. Since 'file' does not exist, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
159 we assume that Crash 3 occurred during the initial creation of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
160 the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
161 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
162 3. Otherwise, use 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
163 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
164 If 'file.tmp' is not present, this is obviously the best we can |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
165 do. This is always the right answer unless Crash 2 occurred, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
166 in which case the contents of 'file' may be newer than they |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
167 were at the point of database backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
168 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
169 If 'file.tmp' is present, we know that we are not actively |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
170 editing the file. The possibilities are: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
171 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
172 a. Crash 1 has occurred. In this case, using 'file' is the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
173 right answer, so we will have chosen correctly. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
174 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
175 b. Crash 3 has occurred. In this case, 'file.tmp' is the right |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
176 answer, so we will have chosen incorrectly. However, 'file' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
177 was at least a previously committed value. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
178 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
179 Future Improvements |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
180 ------------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
181 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
182 One approach would be to take advantage of databases which do |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
183 allow the storage of arbitary date. For example, MySQL provides |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
184 the HUGE BLOB datatype for storing up to 4GB of data. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
185 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
186 Another approach would be to store a version ('v') in the actual |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
187 database and name files 'file.v'. Then, the editing protocol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
188 would become: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
189 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
190 1. Generate a new version 'v', guaranteed to be different from all |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
191 other versions ever used by the database. (The version need |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
192 not be in any particular sequence; a UUID would be fine.) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
193 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
194 2. Store the content in 'file.v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
195 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
196 3. Update the database to indicate that the version of the node is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
197 'v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
198 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
199 Now, if the transaction is committed, the database will refer to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
200 'file.v', where the content exists. If the transaction is rolled |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
201 back, or not committed, 'file.v' will never be referenced. In the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
202 event of a crash, under the assumptions above, there may be |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
203 'file.v' files that are not referenced by the database, but the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
204 database will be consistent, so long as unreferenced 'file.v' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
205 files are never removed until after the database has been backed |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
206 up. |
| 6936 | 207 """ |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
208 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
209 tempext = '.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
210 """The suffix added to files indicating that they are uncommitted.""" |
| 6936 | 211 |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
212 def __init__(self, umask): |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
213 self.umask = umask |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
214 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
215 def subdirFilename(self, classname, nodeid, property=None): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
216 """Determine what the filename and subdir for nodeid + classname is.""" |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
217 if property: |
| 6936 | 218 name = '%s%s.%s' % (classname, nodeid, property) |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
219 else: |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
220 # roundupdb.FileClass never specified the property name, so don't |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
221 # include it |
| 6936 | 222 name = '%s%s' % (classname, nodeid) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
223 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
224 # have a separate subdir for every thousand messages |
|
5400
2120f77554d5
Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4347
diff
changeset
|
225 subdir = str(int(nodeid) // 1000) |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
226 return os.path.join(subdir, name) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
227 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
228 def _tempfile(self, filename): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
229 """Return a temporary filename. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
230 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
231 'filename' -- The name of the eventual destination file.""" |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
232 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
233 return filename + self.tempext |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
234 |
| 4084 | 235 def _editInProgress(self, classname, nodeid, property): |
| 236 """Return true if the file indicated is being edited. | |
| 237 | |
| 238 returns -- True if the current transaction includes an edit to | |
| 239 the file indicated.""" | |
| 240 | |
| 241 for method, args in self.transactions: | |
| 242 if (method == self.doStoreFile and | |
| 6936 | 243 args == (classname, nodeid, property)): |
| 4084 | 244 return True |
| 245 | |
| 246 return False | |
| 247 | |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
248 def filename(self, classname, nodeid, property=None, create=0): |
| 4084 | 249 """Determine what the filename for the given node and optionally |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
250 property is. |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
251 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
252 Try a variety of different filenames - the file could be in the |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
253 usual place, or it could be in a temp file pre-commit *or* it |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
254 could be in an old-style, backwards-compatible flat directory. |
| 4084 | 255 """ |
| 6936 | 256 filename = os.path.join(self.dir, 'files', classname, |
| 257 self.subdirFilename(classname, nodeid, property)) | |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
258 # If the caller is going to create the file, return the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
259 # post-commit filename. It is the callers responsibility to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
260 # add self.tempext when actually creating the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
261 if create: |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
262 return filename |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
263 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
264 tempfile = self._tempfile(filename) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
265 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
266 # If an edit to this file is in progress, then return the name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
267 # of the temporary file containing the edited content. |
| 4084 | 268 if self._editInProgress(classname, nodeid, property): |
| 269 if not os.path.exists(tempfile): | |
| 6936 | 270 raise IOError('content file for %s not found' % tempfile) |
| 4084 | 271 return tempfile |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
272 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
273 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
274 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
275 |
| 6936 | 276 # Otherwise, if the temporary file exists, then the probable |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
277 # explanation is that a crash occurred between the point that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
278 # the database entry recording the creation of the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
279 # occured and the point at which the file was renamed from the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
280 # temporary name to the final name. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
281 if os.path.exists(tempfile): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
282 try: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
283 # Clean up, by performing the commit now. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
284 os.rename(tempfile, filename) |
| 6002 | 285 except OSError: |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
286 pass |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
287 # If two Roundup clients both try to rename the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
288 # at the same time, only one of them will succeed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
289 # So, tolerate such an error -- but no other. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
290 if not os.path.exists(filename): |
| 6936 | 291 raise IOError('content file for %s not found' % filename) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
292 return filename |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
293 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
294 # ok, try flat (very old-style) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
295 if property: |
| 6936 | 296 filename = os.path.join(self.dir, 'files', '%s%s.%s' % ( |
| 297 classname, nodeid, property)) | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
298 else: |
| 6936 | 299 filename = os.path.join(self.dir, 'files', '%s%s' % (classname, |
| 300 nodeid)) | |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
301 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
302 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
303 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
304 # file just ain't there |
| 6936 | 305 raise IOError('content file for %s not found' % filename) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
306 |
|
4347
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
307 def filesize(self, classname, nodeid, property=None, create=0): |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
308 filename = self.filename(classname, nodeid, property, create) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
309 return os.path.getsize(filename) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
310 |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
311 def storefile(self, classname, nodeid, property, content): |
| 4084 | 312 """Store the content of the file in the database. The property may be |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
313 None, in which case the filename does not indicate which property |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
314 is being saved. |
| 4084 | 315 """ |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
316 # determine the name of the file to write to |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
317 name = self.filename(classname, nodeid, property, create=1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
318 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
319 # make sure the file storage dir exists |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
320 if not os.path.exists(os.path.dirname(name)): |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
321 os.makedirs(os.path.dirname(name)) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
322 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
323 # save to a temp file |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
324 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
325 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
326 # make sure we don't register the rename action more than once |
| 4084 | 327 if not self._editInProgress(classname, nodeid, property): |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
328 # save off the rename action |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
329 self.transactions.append((self.doStoreFile, (classname, nodeid, |
| 6936 | 330 property))) |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
331 # always set umask before writing to make sure we have the proper one |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
332 # in multi-tracker (i.e. multi-umask) or modpython scenarios |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
333 # the umask may have changed since last we set it. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
334 os.umask(self.umask) |
|
6486
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
335 fd = open(name, 'wb') |
|
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
336 fd.write(content) |
|
8c371af7e785
Close file in blobfile.py storefile() method.
John Rouillard <rouilj@ieee.org>
parents:
6148
diff
changeset
|
337 fd.close() |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
338 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
339 def getfile(self, classname, nodeid, property): |
| 4084 | 340 """Get the content of the file in the database. |
| 341 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
342 filename = self.filename(classname, nodeid, property) |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
343 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
344 f = open(filename, 'rb') |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
345 try: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
346 # snarf the contents and make sure we close the file |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
347 return f.read() |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
348 finally: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
349 f.close() |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
350 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
351 def numfiles(self): |
| 4084 | 352 """Get number of files in storage, even across subdirectories. |
| 353 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
354 files_dir = os.path.join(self.dir, 'files') |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
355 return files_in_dir(files_dir) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
356 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
357 def doStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 358 """Store the file as part of a transaction commit. |
| 359 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
360 # determine the name of the file to write to |
|
3935
1dab48842cbd
Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents:
3906
diff
changeset
|
361 name = self.filename(classname, nodeid, property, 1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
362 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
363 # the file is currently ".tmp" - move it to its real name to commit |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
364 if name.endswith(self.tempext): |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
365 # creation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
366 dstname = os.path.splitext(name)[0] |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
367 else: |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
368 # edit operation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
369 dstname = name |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
370 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
371 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
372 # content is being updated (and some platforms, eg. win32, won't |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
373 # let us rename over the top of the old file) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
374 if os.path.exists(dstname): |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
375 os.remove(dstname) |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
376 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
377 os.rename(name, dstname) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
378 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
379 # return the classname, nodeid so we reindex this content |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
380 return (classname, nodeid) |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
381 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
382 def rollbackStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 383 """Remove the temp file as a part of a rollback |
| 384 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
385 # determine the name of the file to delete |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
386 name = self.filename(classname, nodeid, property) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
387 if not name.endswith(self.tempext): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
388 name += self.tempext |
|
2966
2ddba486546a
various fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2962
diff
changeset
|
389 os.remove(name) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
390 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
391 def isStoreFile(self, classname, nodeid): |
| 4084 | 392 """See if there is actually any FileStorage for this node. |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
393 Is there a better way than using self.filename? |
| 4084 | 394 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
395 try: |
| 6936 | 396 fname = self.filename(classname, nodeid) # noqa: F841 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
397 return True |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
398 except IOError: |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
399 return False |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
400 |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
401 def destroy(self, classname, nodeid): |
| 4084 | 402 """If there is actually FileStorage for this node |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
403 remove it from the filesystem |
| 4084 | 404 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
405 if self.isStoreFile(classname, nodeid): |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
406 os.remove(self.filename(classname, nodeid)) |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
407 |
|
652
66b324f895d1
add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
650
diff
changeset
|
408 # vim: set filetype=python ts=4 sw=4 et si |