Mercurial > p > roundup > code

annotate roundup/backends/blobfiles.py @ 5470:e2baa4e6ed6d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
handle words starting with unicode characters
author Christof Meerwald <cmeerw@cmeerw.org>
date Sat, 28 Jul 2018 22:39:24 +0100
parents 46317fe544ec
children 6b0c542642be
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
1 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
6 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
11 #
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
17 #
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
18 """This module exports file storage for roundup backends.
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
19 Files are stored into a directory hierarchy.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
20 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1946
diff changeset
21 __docformat__ = 'restructuredtext'
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
22
818
254b8d112eec cleaned up the indexer code:
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
23 import os
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
24
5445
46317fe544ec Python 3 preparation: convert string content to bytes for file storage.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5400
diff changeset
25 from roundup.anypy.strings import s2b
46317fe544ec Python 3 preparation: convert string content to bytes for file storage.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5400
diff changeset
26
2496
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
27 def files_in_dir(dir):
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
28 if not os.path.exists(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
29 return 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
30 num_files = 0
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
31 for dir_entry in os.listdir(dir):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
32 full_filename = os.path.join(dir,dir_entry)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
33 if os.path.isfile(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
34 num_files = num_files + 1
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
35 elif os.path.isdir(full_filename):
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
36 num_files = num_files + files_in_dir(full_filename)
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
37 return num_files
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 645
diff changeset
38
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
39 class FileStorage:
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
40 """Store files in some directory structure
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
41
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
42 Some databases do not permit the storage of arbitrary data (i.e.,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
43 file content). And, some database schema explicitly store file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
44 content in the fielsystem. In particular, if a class defines a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
45 'filename' property, it is assumed that the data is stored in the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
46 indicated file, outside of whatever database Roundup is otherwise
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
47 using.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
48
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
49 In these situations, it is difficult to maintain the transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
50 abstractions used elsewhere in Roundup. In particular, if a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
51 file's content is edited, but then the containing transaction is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
52 not committed, we do not want to commit the edit. Similarly, we
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
53 would like to guarantee that if a transaction is committed to the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
54 database, then the edit has in fact taken place.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
55
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
56 This class provides an approximation of these transactional
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
57 requirements.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
58
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
59 For classes that do not have a 'filename' property, the file name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
60 used to store the file's content is a deterministic function of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
61 the classname and nodeid for the file. The 'filename' function
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
62 computes this name. The name will contain directories and
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
63 subdirectories, but, suppose, for the purposes of what follows,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
64 that the filename is 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
65
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
66 Edit Procotol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
67 -------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
68
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
69 When a file is created or edited, the following protocol is used:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
70
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
71 1. The new content of the file is placed in 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
72
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
73 2. A transaction is recored in 'self.transactions' referencing the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
74 'doStoreFile' method of this class.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
75
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
76 3. At some subsequent point, the database 'commit' function is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
77 called. This function first performs a traditional database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
78 commit (for example, by issuing a SQL command to commit the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
79 current transaction), and, then, runs the transactions recored
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
80 in 'self.transactions'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
81
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
82 4. The 'doStoreFile' method renames the 'file.tmp' to 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
83
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
84 If Step 3 never occurs, but, instead, the database 'rollback'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
85 method is called, then that method, after rolling back the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
86 database transaction, calls 'rollbackStoreFile', which removes
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
87 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
88
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
89 Race Condition
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
90 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
91
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
92 If two Roundup instances (say, the mail gateway and a web client,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
93 or two web clients running with a multi-process server) attempt
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
94 edits at the same time, both will write to 'file.tmp', and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
95 results will be indeterminate.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
96
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
97 Crash Analysis
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
98 --------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
99
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
100 There are several situations that may occur if a crash (whether
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
101 because the machine crashes, because an unhandled Python exception
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
102 is raised, or because the Python process is killed) occurs.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
103
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
104 Complexity ensues because backuping up an RDBMS is generally more
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
105 complex than simply copying a file. Instead, some command is run
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
106 which stores a snapshot of the database in a file. So, if you
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
107 back up the database to a file, and then back up the filesystem,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
108 it is likely that further database transactions have occurred
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
109 between the point of database backup and the point of filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
110 backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
111
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
112 For the purposes, of this analysis, we assume that the filesystem
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
113 backup occurred after the database backup. Furthermore, we assume
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
114 that filesystem backups are atomic; i.e., the at the filesystem is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
115 not being modified during the backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
116
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
117 1. Neither the 'commit' nor 'rollback' methods on the database are
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
118 ever called.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
119
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
120 In this case, the '.tmp' file should be ignored as the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
121 transaction was not committed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
122
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
123 2. The 'commit' method is called. Subsequently, the machine
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
124 crashes, and is restored from backups.
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
125
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
126 The most recent filesystem backup and the most recent database
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
127 backup are not in general from the same instant in time.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
128
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
129 This problem means that we can never be sure after a crash if
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
130 the contents of a file are what we intend. It is always
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
131 possible that an edit was made to the file that is not
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
132 reflected in the filesystem.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
133
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
134 3. A crash occurs between the point of the database commit and the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
135 call to 'doStoreFile'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
136
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
137 If only one of 'file' and 'file.tmp' exists, then that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
138 version should be used. However, if both 'file' and 'file.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
139 exist, there is no way to know which version to use.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
140
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
141 Reading the File
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
142 ----------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
143
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
144 When determining the content of the file, we use the following
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
145 algorithm:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
146
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
147 1. If 'self.transactions' reflects an edit of the file, then use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
148 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
149
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
150 We know that an edit to the file is in process so 'file.tmp' is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
151 the right choice. If 'file.tmp' does not exist, raise an
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
152 exception; something has removed the content of the file while
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
153 we are in the process of editing it.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
154
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
155 2. Otherwise, if 'file.tmp' exists, and 'file' does not, use
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
156 'file.tmp'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
157
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
158 We know that the file is supposed to exist because there is a
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
159 reference to it in the database. Since 'file' does not exist,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
160 we assume that Crash 3 occurred during the initial creation of
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
161 the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
162
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
163 3. Otherwise, use 'file'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
164
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
165 If 'file.tmp' is not present, this is obviously the best we can
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
166 do. This is always the right answer unless Crash 2 occurred,
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
167 in which case the contents of 'file' may be newer than they
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
168 were at the point of database backup.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
169
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
170 If 'file.tmp' is present, we know that we are not actively
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
171 editing the file. The possibilities are:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
172
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
173 a. Crash 1 has occurred. In this case, using 'file' is the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
174 right answer, so we will have chosen correctly.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
175
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
176 b. Crash 3 has occurred. In this case, 'file.tmp' is the right
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
177 answer, so we will have chosen incorrectly. However, 'file'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
178 was at least a previously committed value.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
179
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
180 Future Improvements
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
181 -------------------
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
182
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
183 One approach would be to take advantage of databases which do
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
184 allow the storage of arbitary date. For example, MySQL provides
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
185 the HUGE BLOB datatype for storing up to 4GB of data.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
186
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
187 Another approach would be to store a version ('v') in the actual
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
188 database and name files 'file.v'. Then, the editing protocol
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
189 would become:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
190
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
191 1. Generate a new version 'v', guaranteed to be different from all
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
192 other versions ever used by the database. (The version need
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
193 not be in any particular sequence; a UUID would be fine.)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
194
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
195 2. Store the content in 'file.v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
196
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
197 3. Update the database to indicate that the version of the node is
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
198 'v'.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
199
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
200 Now, if the transaction is committed, the database will refer to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
201 'file.v', where the content exists. If the transaction is rolled
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
202 back, or not committed, 'file.v' will never be referenced. In the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
203 event of a crash, under the assumptions above, there may be
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
204 'file.v' files that are not referenced by the database, but the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
205 database will be consistent, so long as unreferenced 'file.v'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
206 files are never removed until after the database has been backed
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
207 up.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
208 """
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
209
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
210 tempext = '.tmp'
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
211 """The suffix added to files indicating that they are uncommitted."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
212
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
213 def __init__(self, umask):
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
214 self.umask = umask
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
215
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
216 def subdirFilename(self, classname, nodeid, property=None):
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
217 """Determine what the filename and subdir for nodeid + classname is."""
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
218 if property:
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
219 name = '%s%s.%s'%(classname, nodeid, property)
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
220 else:
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
221 # roundupdb.FileClass never specified the property name, so don't
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
222 # include it
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
223 name = '%s%s'%(classname, nodeid)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
224
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
225 # have a separate subdir for every thousand messages
5400
2120f77554d5 Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4347
diff changeset
226 subdir = str(int(nodeid) // 1000)
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
227 return os.path.join(subdir, name)
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
228
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
229 def _tempfile(self, filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
230 """Return a temporary filename.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
231
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
232 'filename' -- The name of the eventual destination file."""
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
233
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
234 return filename + self.tempext
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
235
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
236 def _editInProgress(self, classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
237 """Return true if the file indicated is being edited.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
238
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
239 returns -- True if the current transaction includes an edit to
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
240 the file indicated."""
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
241
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
242 for method, args in self.transactions:
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
243 if (method == self.doStoreFile and
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
244 args == (classname, nodeid, property)):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
245 return True
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
246
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
247 return False
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
248
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
249
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
250 def filename(self, classname, nodeid, property=None, create=0):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
251 """Determine what the filename for the given node and optionally
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
252 property is.
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
253
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
254 Try a variety of different filenames - the file could be in the
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
255 usual place, or it could be in a temp file pre-commit *or* it
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
256 could be in an old-style, backwards-compatible flat directory.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
257 """
3019
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
258 filename = os.path.join(self.dir, 'files', classname,
293a17149765 First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2966
diff changeset
259 self.subdirFilename(classname, nodeid, property))
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
260 # If the caller is going to create the file, return the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
261 # post-commit filename. It is the callers responsibility to
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
262 # add self.tempext when actually creating the file.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
263 if create:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
264 return filename
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
265
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
266 tempfile = self._tempfile(filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
267
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
268 # If an edit to this file is in progress, then return the name
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
269 # of the temporary file containing the edited content.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
270 if self._editInProgress(classname, nodeid, property):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
271 if not os.path.exists(tempfile):
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
272 raise IOError('content file for %s not found'%tempfile)
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
273 return tempfile
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
274
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
275 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
276 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
277
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
278 # Otherwise, if the temporary file exists, then the probable
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
279 # explanation is that a crash occurred between the point that
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
280 # the database entry recording the creation of the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
281 # occured and the point at which the file was renamed from the
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
282 # temporary name to the final name.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
283 if os.path.exists(tempfile):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
284 try:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
285 # Clean up, by performing the commit now.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
286 os.rename(tempfile, filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
287 except:
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
288 pass
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
289 # If two Roundup clients both try to rename the file
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
290 # at the same time, only one of them will succeed.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
291 # So, tolerate such an error -- but no other.
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
292 if not os.path.exists(filename):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
293 raise IOError('content file for %s not found'%filename)
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
294 return filename
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
295
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
296 # ok, try flat (very old-style)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
297 if property:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
298 filename = os.path.join(self.dir, 'files', '%s%s.%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
299 nodeid, property))
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
300 else:
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
301 filename = os.path.join(self.dir, 'files', '%s%s'%(classname,
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
302 nodeid))
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
303 if os.path.exists(filename):
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
304 return filename
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
305
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
306 # file just ain't there
3344
2551a6f4cb78 another name error, another backport candidate
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 3019
diff changeset
307 raise IOError('content file for %s not found'%filename)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
308
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
309 def filesize(self, classname, nodeid, property=None, create=0):
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
310 filename = self.filename(classname, nodeid, property, create)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
311 return os.path.getsize(filename)
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4084
diff changeset
312
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
313 def storefile(self, classname, nodeid, property, content):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
314 """Store the content of the file in the database. The property may be
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
315 None, in which case the filename does not indicate which property
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
316 is being saved.
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
317 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
318 # determine the name of the file to write to
2962
4607f58a007b py2.1 compatibility
Richard Jones <richard@users.sourceforge.net>
parents: 2906
diff changeset
319 name = self.filename(classname, nodeid, property, create=1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
320
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
321 # make sure the file storage dir exists
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
322 if not os.path.exists(os.path.dirname(name)):
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
323 os.makedirs(os.path.dirname(name))
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
324
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
325 # save to a temp file
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
326 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
327
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
328 # make sure we don't register the rename action more than once
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
329 if not self._editInProgress(classname, nodeid, property):
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
330 # save off the rename action
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
331 self.transactions.append((self.doStoreFile, (classname, nodeid,
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
332 property)))
3899
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
333 # always set umask before writing to make sure we have the proper one
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
334 # in multi-tracker (i.e. multi-umask) or modpython scenarios
e02cf6f9ae74 fix class docstring per alex
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
335 # the umask may have changed since last we set it.
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3344
diff changeset
336 os.umask(self.umask)
5445
46317fe544ec Python 3 preparation: convert string content to bytes for file storage.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5400
diff changeset
337 if isinstance(content, str):
46317fe544ec Python 3 preparation: convert string content to bytes for file storage.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5400
diff changeset
338 content = s2b(content)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
339 open(name, 'wb').write(content)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
340
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
341 def getfile(self, classname, nodeid, property):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
342 """Get the content of the file in the database.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
343 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
344 filename = self.filename(classname, nodeid, property)
2899
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
345
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
346 f = open(filename, 'rb')
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
347 try:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
348 # snarf the contents and make sure we close the file
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
349 return f.read()
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
350 finally:
09a4d6dd6dcb Handle older (really older) anydbm databases in export code.
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
351 f.close()
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
352
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
353 def numfiles(self):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
354 """Get number of files in storage, even across subdirectories.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
355 """
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
356 files_dir = os.path.join(self.dir, 'files')
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
357 return files_in_dir(files_dir)
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
358
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
359 def doStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
360 """Store the file as part of a transaction commit.
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
361 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
362 # determine the name of the file to write to
3935
1dab48842cbd Throwing up hands in resignation and finally deleting the metakit backend.
Richard Jones <richard@users.sourceforge.net>
parents: 3906
diff changeset
363 name = self.filename(classname, nodeid, property, 1)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
364
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
365 # the file is currently ".tmp" - move it to its real name to commit
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
366 if name.endswith(self.tempext):
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
367 # creation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
368 dstname = os.path.splitext(name)[0]
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
369 else:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
370 # edit operation
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
371 dstname = name
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
372 name = self._tempfile(name)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
373
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
374 # content is being updated (and some platforms, eg. win32, won't
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
375 # let us rename over the top of the old file)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
376 if os.path.exists(dstname):
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
377 os.remove(dstname)
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
378
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
379 os.rename(name, dstname)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
380
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
381 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
382 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
383
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 864
diff changeset
384 def rollbackStoreFile(self, classname, nodeid, property, **databases):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
385 """Remove the temp file as a part of a rollback
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
386 """
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
387 # determine the name of the file to delete
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
388 name = self.filename(classname, nodeid, property)
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
389 if not name.endswith(self.tempext):
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3935
diff changeset
390 name += self.tempext
2966
2ddba486546a various fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2962
diff changeset
391 os.remove(name)
645
e6d1c6d66de3 add module blobfiles in backends with file access functions. not yet activated.
Engelbert Gruber <grubert@users.sourceforge.net>
parents:
diff changeset
392
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
393 def isStoreFile(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
394 """See if there is actually any FileStorage for this node.
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
395 Is there a better way than using self.filename?
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
396 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
397 try:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
398 fname = self.filename(classname, nodeid)
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
399 return True
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
400 except IOError:
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
401 return False
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
402
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
403 def destroy(self, classname, nodeid):
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
404 """If there is actually FileStorage for this node
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
405 remove it from the filesystem
4084
35d2976cb4ba Fix issue2550510
Stefan Seefeld <stefan@seefeld.name>
parents: 3960
diff changeset
406 """
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
407 if self.isStoreFile(classname, nodeid):
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
408 os.remove(self.filename(classname, nodeid))
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3899
diff changeset
409
652
66b324f895d1 add, vim line and cvs log key.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 650
diff changeset
410 # vim: set filetype=python ts=4 sw=4 et si
Roundup Issue Tracker: http://roundup-tracker.org/