Mercurial > p > roundup > code

annotate roundup/backends/sessions_dbm.py @ 6565:2c2dbfc332ba

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Try to handle multiple connections better. The session database is a hot spot. When multiple requests (e.g. 20) come in at the same time session database contention can get great. The original code didn't retry session database access when the open failed. This resulted in errors at the client. The second pass delayed 0.01 seconds and retried. It was better but we still had multiple second stalls. I think the first request got in, everybody else backed up and then retried at the same time. Again they stepped on each other. With logging I would see many counters go all the way to low single digits or to -1 indicating falure. This pass uses randomint to generate delays from 0-.125 seconds in 5ms increments. This performs better in testing. I rarely saw a counter less than 13 (2 failed retries). Current logging starts after 6 failures and counts down until success or failure.
author John Rouillard <rouilj@ieee.org>
date Thu, 16 Dec 2021 20:02:00 -0500
parents bef1e42be04c
children b4d0b48b3096
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1 """This module defines a very basic store that's used by the CGI interface
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2 to store session and one-time-key information.
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
4 Yes, it's called "sessions" - because originally it only defined a session
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
5 class. It's now also used for One Time Key handling too.
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
6 """
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
7 __docformat__ = 'restructuredtext'
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
8
6565
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
9 import os, marshal, time, logging, random
4362
74476eaac38a more modernisation
Richard Jones <richard@users.sourceforge.net>
parents: 3989
diff changeset
10
5837
883c9e90b403 Fix problem with cgi.escape being depricated a different way. This way
John Rouillard <rouilj@ieee.org>
parents: 5806
diff changeset
11 from roundup.anypy.html import html_escape as escape
5806
abee2c2c822e More cgi.escape/html.escape fixes.
John Rouillard <rouilj@ieee.org>
parents: 5732
diff changeset
12
3924
21d3d7eeea8c assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3609
diff changeset
13 from roundup import hyperdb
3925
603ec9630b08 i18n for hyperdb and backend errors
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3924
diff changeset
14 from roundup.i18n import _
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
15 from roundup.anypy.dbm_ import anydbm, whichdb
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
16
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
17
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
18 class BasicDatabase:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
19 ''' Provide a nice encapsulation of an anydbm store.
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
20
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
21 Keys are id strings, values are automatically marshalled data.
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
22 '''
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
23 _db_type = None
5252
0b154486ed38 Make sure that the name property is initialized to none.
John Rouillard <rouilj@ieee.org>
parents: 5011
diff changeset
24 name = None
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
25
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
26 def __init__(self, db):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
27 self.config = db.config
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
28 self.dir = db.config.DATABASE
3609
f2fda3e6fc8b umask is now configurable (with the same 0002 default)
Richard Jones <richard@users.sourceforge.net>
parents: 3606
diff changeset
29 os.umask(db.config.UMASK)
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
30
2169
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
31 def exists(self, infoid):
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
32 db = self.opendb('c')
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
33 try:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
34 return infoid in db
2169
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
35 finally:
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
36 db.close()
12cd4fa91eb7 OTK generation was busted (thanks Stuart D. Gathman)
Richard Jones <richard@users.sourceforge.net>
parents: 2151
diff changeset
37
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
38 def clear(self):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39 path = os.path.join(self.dir, self.name)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
40 if os.path.exists(path):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
41 os.remove(path)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
42 elif os.path.exists(path+'.db'): # dbm appends .db
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
43 os.remove(path+'.db')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
44
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
45 def cache_db_type(self, path):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
46 ''' determine which DB wrote the class file, and cache it as an
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
47 attribute of __class__ (to allow for subclassed DBs to be
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
48 different sorts)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
49 '''
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
50 db_type = ''
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
51 if os.path.exists(path):
4363
Richard Jones <richard@users.sourceforge.net>
parents: 4362
diff changeset
52 db_type = whichdb(path)
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
53 if not db_type:
4362
74476eaac38a more modernisation
Richard Jones <richard@users.sourceforge.net>
parents: 3989
diff changeset
54 raise hyperdb.DatabaseError(
74476eaac38a more modernisation
Richard Jones <richard@users.sourceforge.net>
parents: 3989
diff changeset
55 _("Couldn't identify database type"))
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
56 elif os.path.exists(path+'.db'):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
57 # if the path ends in '.db', it's a dbm database, whether
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
58 # anydbm says it's dbhash or not!
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
59 db_type = 'dbm'
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
60 self.__class__._db_type = db_type
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
61
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
62 _marker = []
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
63
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 def get(self, infoid, value, default=_marker):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
65 db = self.opendb('c')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
66 try:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
67 if infoid in db:
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
68 values = marshal.loads(db[infoid])
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
69 else:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
70 if default != self._marker:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
71 return default
6036
5eeb073e33c7 flake8 whitespace fixes, remove unused exception var.
John Rouillard <rouilj@ieee.org>
parents: 5837
diff changeset
72 raise KeyError('No such %s "%s"' % (self.name, escape(infoid)))
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
73 return values.get(value, None)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
74 finally:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75 db.close()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
77 def getall(self, infoid):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
78 db = self.opendb('c')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
79 try:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
80 try:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
81 d = marshal.loads(db[infoid])
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
82 del d['__timestamp']
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
83 return d
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
84 except KeyError:
6036
5eeb073e33c7 flake8 whitespace fixes, remove unused exception var.
John Rouillard <rouilj@ieee.org>
parents: 5837
diff changeset
85 raise KeyError('No such %s "%s"' % (self.name, escape(infoid)))
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
86 finally:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
87 db.close()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
88
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
89 def set(self, infoid, **newvalues):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
90 db = self.opendb('c')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
91 try:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
92 if infoid in db:
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
93 values = marshal.loads(db[infoid])
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
94 else:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
95 values = {'__timestamp': time.time()}
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
96 values.update(newvalues)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
97 db[infoid] = marshal.dumps(values)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
98 finally:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
99 db.close()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
100
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
101 def list(self):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
102 db = self.opendb('r')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
103 try:
4385
8d135c19523e fix up some pre-Python2.6 compatibility issues in the *dbm interface
Richard Jones <richard@users.sourceforge.net>
parents: 4383
diff changeset
104 return list(db.keys())
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
105 finally:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106 db.close()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 def destroy(self, infoid):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 db = self.opendb('c')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110 try:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4585
diff changeset
111 if infoid in db:
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 del db[infoid]
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 finally:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
114 db.close()
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
115
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
116 def opendb(self, mode):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
117 '''Low-level database opener that gets around anydbm/dbm
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
118 eccentricities.
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
119 '''
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
120 # figure the class db type
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 path = os.path.join(os.getcwd(), self.dir, self.name)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
122 if self._db_type is None:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
123 self.cache_db_type(path)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
124
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
125 db_type = self._db_type
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
126
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
127 # new database? let anydbm pick the best dbm
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
128 if not db_type:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
129 return anydbm.open(path, 'c')
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
130
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
131 # open the database with the correct module
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
132 dbm = __import__(db_type)
5732
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
133
6036
5eeb073e33c7 flake8 whitespace fixes, remove unused exception var.
John Rouillard <rouilj@ieee.org>
parents: 5837
diff changeset
134 retries_left = 15
6565
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
135 logger = logging.getLogger('roundup.hyperdb.backend.sessions')
5732
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
136 while True:
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
137 try:
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
138 handle = dbm.open(path, mode)
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
139 break
6565
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
140 except OSError as e:
5732
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
141 # Primarily we want to catch and retry:
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
142 # [Errno 11] Resource temporarily unavailable retry
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
143 # FIXME: make this more specific
6565
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
144 if retries_left < 10:
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
145 logger.warning('dbm.open failed, retrying %s left: %s'%(retries_left,e))
5732
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
146 if retries_left < 0:
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
147 # We have used up the retries. Reraise the exception
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
148 # that got us here.
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
149 raise
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
150 else:
6565
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
151 # stagger retry to try to get around thundering herd issue.
2c2dbfc332ba Try to handle multiple connections better.
John Rouillard <rouilj@ieee.org>
parents: 6064
diff changeset
152 time.sleep(random.randint(0,25)*.005)
6064
bef1e42be04c Flake8 whitespace changes.
John Rouillard <rouilj@ieee.org>
parents: 6036
diff changeset
153 retries_left = retries_left - 1
5732
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
154 continue # the while loop
0e6ed3d72f92 Rest rate limiting code first commit. It is a bit rough and turned off
John Rouillard <rouilj@ieee.org>
parents: 5252
diff changeset
155 return handle
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
156
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
157 def commit(self):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
158 pass
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
159
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
160 def close(self):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
161 pass
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
162
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
163 def updateTimestamp(self, sessid):
3606
04dc3eef67b7 reduced frequency of session timestamp update
Richard Jones <richard@users.sourceforge.net>
parents: 2169
diff changeset
164 ''' don't update every hit - once a minute should be OK '''
04dc3eef67b7 reduced frequency of session timestamp update
Richard Jones <richard@users.sourceforge.net>
parents: 2169
diff changeset
165 sess = self.get(sessid, '__timestamp', None)
04dc3eef67b7 reduced frequency of session timestamp update
Richard Jones <richard@users.sourceforge.net>
parents: 2169
diff changeset
166 now = time.time()
04dc3eef67b7 reduced frequency of session timestamp update
Richard Jones <richard@users.sourceforge.net>
parents: 2169
diff changeset
167 if sess is None or now > sess + 60:
04dc3eef67b7 reduced frequency of session timestamp update
Richard Jones <richard@users.sourceforge.net>
parents: 2169
diff changeset
168 self.set(sessid, __timestamp=now)
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
169
3989
0112e9e1d068 improvements to session management
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
170 def clean(self):
0112e9e1d068 improvements to session management
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
171 ''' Remove session records that haven't been used for a week. '''
0112e9e1d068 improvements to session management
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
172 now = time.time()
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
173 week = 60*60*24*7
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
174 for sessid in self.list():
2151
ae21c1fc41d5 better fix
Richard Jones <richard@users.sourceforge.net>
parents: 2150
diff changeset
175 sess = self.get(sessid, '__timestamp', None)
ae21c1fc41d5 better fix
Richard Jones <richard@users.sourceforge.net>
parents: 2150
diff changeset
176 if sess is None:
ae21c1fc41d5 better fix
Richard Jones <richard@users.sourceforge.net>
parents: 2150
diff changeset
177 self.updateTimestamp(sessid)
3989
0112e9e1d068 improvements to session management
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
178 continue
2151
ae21c1fc41d5 better fix
Richard Jones <richard@users.sourceforge.net>
parents: 2150
diff changeset
179 interval = now - sess
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
180 if interval > week:
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
181 self.destroy(sessid)
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
182
6064
bef1e42be04c Flake8 whitespace changes.
John Rouillard <rouilj@ieee.org>
parents: 6036
diff changeset
183
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
184 class Sessions(BasicDatabase):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
185 name = 'sessions'
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
186
6064
bef1e42be04c Flake8 whitespace changes.
John Rouillard <rouilj@ieee.org>
parents: 6036
diff changeset
187
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
188 class OneTimeKeys(BasicDatabase):
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
189 name = 'otks'
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
190
3925
603ec9630b08 i18n for hyperdb and backend errors
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3924
diff changeset
191 # vim: set sts ts=4 sw=4 et si :
Roundup Issue Tracker: http://roundup-tracker.org/