Mercurial > p > roundup > code
annotate roundup/backends/blobfiles.py @ 5548:fea11d05110e
Avoid errors from selecting "no selection" on multilink (issue2550722).
As discussed in issue 2550722 there are various cases where selecting
"no selection" on a multilink can result in inappropriate errors from
Roundup:
* If selecting "no selection" produces a null edit (a value was set in
the multilink in an edit with an error, then removed again, along
with all other changes, in the next form submission), so the page is
rendered from the form contents including the "-<id>" value for "no
selection" for the multilink.
* If creating an item with a nonempty value for a multilink has an
error, and the resubmission changes that multilink to "no selection"
(and this in turn has subcases, according to whether the creation
then succeeds or fails on the resubmission, which need fixes in
different places in the Roundup code).
All of these cases have in common that it is expected and OK to have a
"-<id>" value for a submission for a multilink when <id> is not set in
that multilink in the database (because the original attempt to set
<id> in that multilink had an error), so the hyperdb.py logic to give
an error in that case is thus removed. In the subcase of the second
case where the resubmission with "no selection" has an error, the
templating code tries to produce a menu entry for the "-<id>"
multilink value, which also results in an error, hence the
templating.py change to ignore such values in the list for a
multilink.
| author | Joseph Myers <jsm@polyomino.org.uk> |
|---|---|
| date | Thu, 27 Sep 2018 11:33:01 +0000 |
| parents | 6b0c542642be |
| children | 3175bb92ca28 |
| rev | line source |
|---|---|
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
1 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
6 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
11 # |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
17 # |
| 4084 | 18 """This module exports file storage for roundup backends. |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
19 Files are stored into a directory hierarchy. |
| 4084 | 20 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1946
diff
changeset
|
21 __docformat__ = 'restructuredtext' |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
22 |
|
818
254b8d112eec
cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents:
778
diff
changeset
|
23 import os |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
24 |
| 2496 | 25 def files_in_dir(dir): |
|
650
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
26 if not os.path.exists(dir): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
27 return 0 |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
28 num_files = 0 |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
29 for dir_entry in os.listdir(dir): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
30 full_filename = os.path.join(dir,dir_entry) |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
31 if os.path.isfile(full_filename): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
32 num_files = num_files + 1 |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
33 elif os.path.isdir(full_filename): |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
34 num_files = num_files + files_in_dir(full_filename) |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
35 return num_files |
|
9b2575610953
Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents:
645
diff
changeset
|
36 |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
37 class FileStorage: |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
38 """Store files in some directory structure |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
39 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
40 Some databases do not permit the storage of arbitrary data (i.e., |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
41 file content). And, some database schema explicitly store file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
42 content in the fielsystem. In particular, if a class defines a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
43 'filename' property, it is assumed that the data is stored in the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
44 indicated file, outside of whatever database Roundup is otherwise |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
45 using. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
46 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
47 In these situations, it is difficult to maintain the transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
48 abstractions used elsewhere in Roundup. In particular, if a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
49 file's content is edited, but then the containing transaction is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
50 not committed, we do not want to commit the edit. Similarly, we |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
51 would like to guarantee that if a transaction is committed to the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
52 database, then the edit has in fact taken place. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
53 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
54 This class provides an approximation of these transactional |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
55 requirements. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
56 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
57 For classes that do not have a 'filename' property, the file name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
58 used to store the file's content is a deterministic function of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
59 the classname and nodeid for the file. The 'filename' function |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
60 computes this name. The name will contain directories and |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
61 subdirectories, but, suppose, for the purposes of what follows, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
62 that the filename is 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
63 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
64 Edit Procotol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
65 ------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
66 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
67 When a file is created or edited, the following protocol is used: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
68 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
69 1. The new content of the file is placed in 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
70 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
71 2. A transaction is recored in 'self.transactions' referencing the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
72 'doStoreFile' method of this class. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
73 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
74 3. At some subsequent point, the database 'commit' function is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
75 called. This function first performs a traditional database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
76 commit (for example, by issuing a SQL command to commit the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
77 current transaction), and, then, runs the transactions recored |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
78 in 'self.transactions'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
79 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
80 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
81 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
82 If Step 3 never occurs, but, instead, the database 'rollback' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
83 method is called, then that method, after rolling back the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
84 database transaction, calls 'rollbackStoreFile', which removes |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
85 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
86 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
87 Race Condition |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
88 -------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
89 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
90 If two Roundup instances (say, the mail gateway and a web client, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
91 or two web clients running with a multi-process server) attempt |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
92 edits at the same time, both will write to 'file.tmp', and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
93 results will be indeterminate. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
94 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
95 Crash Analysis |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
96 -------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
97 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
98 There are several situations that may occur if a crash (whether |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
99 because the machine crashes, because an unhandled Python exception |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
100 is raised, or because the Python process is killed) occurs. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
101 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
102 Complexity ensues because backuping up an RDBMS is generally more |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
103 complex than simply copying a file. Instead, some command is run |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
104 which stores a snapshot of the database in a file. So, if you |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
105 back up the database to a file, and then back up the filesystem, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
106 it is likely that further database transactions have occurred |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
107 between the point of database backup and the point of filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
108 backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
109 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
110 For the purposes, of this analysis, we assume that the filesystem |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
111 backup occurred after the database backup. Furthermore, we assume |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
112 that filesystem backups are atomic; i.e., the at the filesystem is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
113 not being modified during the backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
114 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
115 1. Neither the 'commit' nor 'rollback' methods on the database are |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
116 ever called. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
117 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
118 In this case, the '.tmp' file should be ignored as the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
119 transaction was not committed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
120 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
121 2. The 'commit' method is called. Subsequently, the machine |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
122 crashes, and is restored from backups. |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
123 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
124 The most recent filesystem backup and the most recent database |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
125 backup are not in general from the same instant in time. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
126 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
127 This problem means that we can never be sure after a crash if |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
128 the contents of a file are what we intend. It is always |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
129 possible that an edit was made to the file that is not |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
130 reflected in the filesystem. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
131 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
132 3. A crash occurs between the point of the database commit and the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
133 call to 'doStoreFile'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
134 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
135 If only one of 'file' and 'file.tmp' exists, then that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
136 version should be used. However, if both 'file' and 'file.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
137 exist, there is no way to know which version to use. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
138 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
139 Reading the File |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
140 ---------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
141 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
142 When determining the content of the file, we use the following |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
143 algorithm: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
144 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
145 1. If 'self.transactions' reflects an edit of the file, then use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
146 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
147 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
148 We know that an edit to the file is in process so 'file.tmp' is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
149 the right choice. If 'file.tmp' does not exist, raise an |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
150 exception; something has removed the content of the file while |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
151 we are in the process of editing it. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
152 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
153 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
154 'file.tmp'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
155 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
156 We know that the file is supposed to exist because there is a |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
157 reference to it in the database. Since 'file' does not exist, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
158 we assume that Crash 3 occurred during the initial creation of |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
159 the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
160 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
161 3. Otherwise, use 'file'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
162 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
163 If 'file.tmp' is not present, this is obviously the best we can |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
164 do. This is always the right answer unless Crash 2 occurred, |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
165 in which case the contents of 'file' may be newer than they |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
166 were at the point of database backup. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
167 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
168 If 'file.tmp' is present, we know that we are not actively |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
169 editing the file. The possibilities are: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
170 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
171 a. Crash 1 has occurred. In this case, using 'file' is the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
172 right answer, so we will have chosen correctly. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
173 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
174 b. Crash 3 has occurred. In this case, 'file.tmp' is the right |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
175 answer, so we will have chosen incorrectly. However, 'file' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
176 was at least a previously committed value. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
177 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
178 Future Improvements |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
179 ------------------- |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
180 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
181 One approach would be to take advantage of databases which do |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
182 allow the storage of arbitary date. For example, MySQL provides |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
183 the HUGE BLOB datatype for storing up to 4GB of data. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
184 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
185 Another approach would be to store a version ('v') in the actual |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
186 database and name files 'file.v'. Then, the editing protocol |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
187 would become: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
188 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
189 1. Generate a new version 'v', guaranteed to be different from all |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
190 other versions ever used by the database. (The version need |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
191 not be in any particular sequence; a UUID would be fine.) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
192 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
193 2. Store the content in 'file.v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
194 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
195 3. Update the database to indicate that the version of the node is |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
196 'v'. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
197 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
198 Now, if the transaction is committed, the database will refer to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
199 'file.v', where the content exists. If the transaction is rolled |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
200 back, or not committed, 'file.v' will never be referenced. In the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
201 event of a crash, under the assumptions above, there may be |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
202 'file.v' files that are not referenced by the database, but the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
203 database will be consistent, so long as unreferenced 'file.v' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
204 files are never removed until after the database has been backed |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
205 up. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
206 """ |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
207 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
208 tempext = '.tmp' |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
209 """The suffix added to files indicating that they are uncommitted.""" |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
210 |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
211 def __init__(self, umask): |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
212 self.umask = umask |
|
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
213 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
214 def subdirFilename(self, classname, nodeid, property=None): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
215 """Determine what the filename and subdir for nodeid + classname is.""" |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
216 if property: |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
217 name = '%s%s.%s'%(classname, nodeid, property) |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
218 else: |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
219 # roundupdb.FileClass never specified the property name, so don't |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
220 # include it |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
221 name = '%s%s'%(classname, nodeid) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
222 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
223 # have a separate subdir for every thousand messages |
|
5400
2120f77554d5
Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4347
diff
changeset
|
224 subdir = str(int(nodeid) // 1000) |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
225 return os.path.join(subdir, name) |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
226 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
227 def _tempfile(self, filename): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
228 """Return a temporary filename. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
229 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
230 'filename' -- The name of the eventual destination file.""" |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
231 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
232 return filename + self.tempext |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
233 |
| 4084 | 234 def _editInProgress(self, classname, nodeid, property): |
| 235 """Return true if the file indicated is being edited. | |
| 236 | |
| 237 returns -- True if the current transaction includes an edit to | |
| 238 the file indicated.""" | |
| 239 | |
| 240 for method, args in self.transactions: | |
| 241 if (method == self.doStoreFile and | |
| 242 args == (classname, nodeid, property)): | |
| 243 return True | |
| 244 | |
| 245 return False | |
| 246 | |
| 247 | |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
248 def filename(self, classname, nodeid, property=None, create=0): |
| 4084 | 249 """Determine what the filename for the given node and optionally |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
250 property is. |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
251 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
252 Try a variety of different filenames - the file could be in the |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
253 usual place, or it could be in a temp file pre-commit *or* it |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
254 could be in an old-style, backwards-compatible flat directory. |
| 4084 | 255 """ |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
256 filename = os.path.join(self.dir, 'files', classname, |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2966
diff
changeset
|
257 self.subdirFilename(classname, nodeid, property)) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
258 # If the caller is going to create the file, return the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
259 # post-commit filename. It is the callers responsibility to |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
260 # add self.tempext when actually creating the file. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
261 if create: |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
262 return filename |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
263 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
264 tempfile = self._tempfile(filename) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
265 |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
266 # If an edit to this file is in progress, then return the name |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
267 # of the temporary file containing the edited content. |
| 4084 | 268 if self._editInProgress(classname, nodeid, property): |
| 269 if not os.path.exists(tempfile): | |
| 270 raise IOError('content file for %s not found'%tempfile) | |
| 271 return tempfile | |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
272 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
273 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
274 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
275 |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
276 # Otherwise, if the temporary file exists, then the probable |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
277 # explanation is that a crash occurred between the point that |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
278 # the database entry recording the creation of the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
279 # occured and the point at which the file was renamed from the |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
280 # temporary name to the final name. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
281 if os.path.exists(tempfile): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
282 try: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
283 # Clean up, by performing the commit now. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
284 os.rename(tempfile, filename) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
285 except: |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
286 pass |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
287 # If two Roundup clients both try to rename the file |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
288 # at the same time, only one of them will succeed. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
289 # So, tolerate such an error -- but no other. |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
290 if not os.path.exists(filename): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
291 raise IOError('content file for %s not found'%filename) |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
292 return filename |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
293 |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
294 # ok, try flat (very old-style) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
295 if property: |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
296 filename = os.path.join(self.dir, 'files', '%s%s.%s'%(classname, |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
297 nodeid, property)) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
298 else: |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
299 filename = os.path.join(self.dir, 'files', '%s%s'%(classname, |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
300 nodeid)) |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
301 if os.path.exists(filename): |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
302 return filename |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
303 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
304 # file just ain't there |
|
3344
2551a6f4cb78
another name error, another backport candidate
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
3019
diff
changeset
|
305 raise IOError('content file for %s not found'%filename) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
306 |
|
4347
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
307 def filesize(self, classname, nodeid, property=None, create=0): |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
308 filename = self.filename(classname, nodeid, property, create) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
309 return os.path.getsize(filename) |
|
0e33bf5571dc
make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents:
4084
diff
changeset
|
310 |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
311 def storefile(self, classname, nodeid, property, content): |
| 4084 | 312 """Store the content of the file in the database. The property may be |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
313 None, in which case the filename does not indicate which property |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
314 is being saved. |
| 4084 | 315 """ |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
316 # determine the name of the file to write to |
|
2962
4607f58a007b
py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents:
2906
diff
changeset
|
317 name = self.filename(classname, nodeid, property, create=1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
318 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
319 # make sure the file storage dir exists |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
320 if not os.path.exists(os.path.dirname(name)): |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
321 os.makedirs(os.path.dirname(name)) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
322 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
323 # save to a temp file |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
324 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
325 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
326 # make sure we don't register the rename action more than once |
| 4084 | 327 if not self._editInProgress(classname, nodeid, property): |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
328 # save off the rename action |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
329 self.transactions.append((self.doStoreFile, (classname, nodeid, |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
330 property))) |
|
3899
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
331 # always set umask before writing to make sure we have the proper one |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
332 # in multi-tracker (i.e. multi-umask) or modpython scenarios |
|
e02cf6f9ae74
fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
333 # the umask may have changed since last we set it. |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3344
diff
changeset
|
334 os.umask(self.umask) |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
335 open(name, 'wb').write(content) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
336 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
337 def getfile(self, classname, nodeid, property): |
| 4084 | 338 """Get the content of the file in the database. |
| 339 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
340 filename = self.filename(classname, nodeid, property) |
|
2899
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
341 |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
342 f = open(filename, 'rb') |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
343 try: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
344 # snarf the contents and make sure we close the file |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
345 return f.read() |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
346 finally: |
|
09a4d6dd6dcb
Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
347 f.close() |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
348 |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
349 def numfiles(self): |
| 4084 | 350 """Get number of files in storage, even across subdirectories. |
| 351 """ | |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
352 files_dir = os.path.join(self.dir, 'files') |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
353 return files_in_dir(files_dir) |
|
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
354 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
355 def doStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 356 """Store the file as part of a transaction commit. |
| 357 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
358 # determine the name of the file to write to |
|
3935
1dab48842cbd
Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents:
3906
diff
changeset
|
359 name = self.filename(classname, nodeid, property, 1) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
360 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
361 # the file is currently ".tmp" - move it to its real name to commit |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
362 if name.endswith(self.tempext): |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
363 # creation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
364 dstname = os.path.splitext(name)[0] |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
365 else: |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
366 # edit operation |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
367 dstname = name |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
368 name = self._tempfile(name) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
369 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
370 # content is being updated (and some platforms, eg. win32, won't |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
371 # let us rename over the top of the old file) |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
372 if os.path.exists(dstname): |
|
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
373 os.remove(dstname) |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
374 |
|
2906
a8808157f892
fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents:
2899
diff
changeset
|
375 os.rename(name, dstname) |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
376 |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
377 # return the classname, nodeid so we reindex this content |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
378 return (classname, nodeid) |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
379 |
|
891
974a4b94c5e3
Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents:
864
diff
changeset
|
380 def rollbackStoreFile(self, classname, nodeid, property, **databases): |
| 4084 | 381 """Remove the temp file as a part of a rollback |
| 382 """ | |
|
825
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
383 # determine the name of the file to delete |
|
0779ea9f1f18
More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents:
818
diff
changeset
|
384 name = self.filename(classname, nodeid, property) |
|
3960
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
385 if not name.endswith(self.tempext): |
|
bc412bb2ccd3
Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents:
3935
diff
changeset
|
386 name += self.tempext |
|
2966
2ddba486546a
various fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2962
diff
changeset
|
387 os.remove(name) |
|
645
e6d1c6d66de3
add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff
changeset
|
388 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
389 def isStoreFile(self, classname, nodeid): |
| 4084 | 390 """See if there is actually any FileStorage for this node. |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
391 Is there a better way than using self.filename? |
| 4084 | 392 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
393 try: |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
394 fname = self.filename(classname, nodeid) |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
395 return True |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
396 except IOError: |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
397 return False |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
398 |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
399 def destroy(self, classname, nodeid): |
| 4084 | 400 """If there is actually FileStorage for this node |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
401 remove it from the filesystem |
| 4084 | 402 """ |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
403 if self.isStoreFile(classname, nodeid): |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
404 os.remove(self.filename(classname, nodeid)) |
|
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3899
diff
changeset
|
405 |
|
652
66b324f895d1
add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
650
diff
changeset
|
406 # vim: set filetype=python ts=4 sw=4 et si |
