Mercurial > p > roundup > code

annotate roundup/backends/sessions_redis.py @ 6814:3f60a71b0812

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Summary: Support selecion session/otk data store. Add redis as data store. Allow admin to select the backend data store. Compatibility matrix: main\/ session>| anydbm | sqlite | redis | mysql | postgresql | anydbm | D | | X | | | sqlite | X | D | X | | | mysql | | | | D | | postgresql | | | | | D | --------------------------------------------------------------+ D - default if unconfigured, X - compatible choice DETAILS roundup/configuration.py: add config.ini section sessiondb with settings: backend and redis_url. CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt: doc on config of session db and redis. Plus some other fixes: admin - clarified why we do not drop __words and __testids table in native-fts conversion. TYpo fix. upgrading - doc how you can keep using anydbm for session data with sqlite. Fix dupe sentence in an upgrading config.ini section. roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py: code to support redis, redis/anydbm backends respectively. roundup/backends/sessions_redis.py new storage backend for redis. roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py redis uses a different way of calculating lifetime/timestamp. Since expiration of an item occurred if its timestamp was more than 1 week old, code would calculate: now - 1 week + lifetime. But this results in faster expiration in redis if used for lifetime/timestamp. Convert code to use the lifetime() method in BasicDatabase that generates the right timestamp for each backend. test/session_common.py: added tests for more cases, get without default, getall non-existing key etc. timestamp test changed to use new self.get_ts which is overridden in other tests. Test that datatypes survive storage. test/test_redis_session.py: test redis session store with sqlite and anydbm primary databases test/test_anydbm.py, test/test_sqlite.py add test to make sure the databases are properly set up sqlite - add test cases where anydbm is used as datastore anydbm - remove updateTimestamp override add get_ts(). test/test_config.py tests on redis_url and compatibility on choice of sessiondb backend .travis.yml: add redis db and redis-py
author John Rouillard <rouilj@ieee.org>
date Thu, 04 Aug 2022 14:41:58 -0400
parents
children fe0091279f50
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 """This module defines a redis based store that's used by
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 the CGI interface to store session and one-time-key
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 information.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 Yes, it's called "sessions" - because originally it only
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6 defined a session class. It's now also used for One Time Key
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 handling too.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 It uses simple strings rather than redis hash structure
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 because the hash the values are always strings. We need to
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 be able to represent the same data types available to rdbms
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 and dbm session stores.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14 session_dbm uses marshal.dumps and marshal.loads. This seems
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 4 or 18 times faster than the repr()/eval() used by
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 session_rdbms. So use marshal even though it is impossible
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 to read when viewing (using redis-cli).
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18 """
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19 __docformat__ = 'restructuredtext'
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 import logging, marshal, redis, time
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 from roundup.anypy.html import html_escape as escape
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 from roundup.i18n import _
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 class BasicDatabase:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 ''' Provide a nice encapsulation of a redis store.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 Keys are id strings, values are automatically marshalled data.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 name = None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 default_lifetime = 60*60*24*7 # 1 week
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # FIXME: figure out how to allow admin to change this
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 # to repr/eval using interfaces.py or other method.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 # marshalled data is not readable when debugging.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 tostr = marshal.dumps
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 todict = marshal.loads
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42 def __init__(self, db):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 self.config = db.config
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 url = self.config.SESSIONDB_REDIS_URL
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46 # Example at default port without auth.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
47 # redis://localhost:6379/0?health_check_interval=2
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48 #
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 # Do not allow decode_responses=True in url, data is
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
50 # marshal'ed binary data that will get broken by decoding.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 # Enforce this in configuration.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 self.redis = redis.Redis.from_url(url=url, decode_responses=False)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 def makekey(self, key):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 '''method to namespace all keys using self.name:....'''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 return "%s:%s" % (self.name, key)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 def exists(self, infoid):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 return self.redis.exists(self.makekey(infoid))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 def clear(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 '''Delete all keys from the database.'''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 self.redis.flushdb()
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65 _marker = []
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67 def get(self, infoid, value, default=_marker):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 '''get a specific value from the data associated with a key'''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 infoid = self.makekey(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 v = self.redis.get(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 if not v:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 if default != self._marker:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 return default
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 raise KeyError(_('Key %(key)s not found in %(name)s '
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 'database.' % {"name": self.name,
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 "key": escape(infoid)}))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 return self.todict(v)[value]
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79 def getall(self, infoid):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 '''return all values associated with a key'''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 d = self.redis.get(self.makekey(infoid))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
83 if d is not None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
84 d = self.todict(d)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
85 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
86 d = {}
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
87 del d['__timestamp']
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
88 return d
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89 except KeyError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 # It is possible for d to be malformed missing __timestamp.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 # If so, we get a misleading error, but anydbm does the
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92 # same so....
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
93 raise KeyError(_('Key %(key)s not found in %(name)s '
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
94 'database.' % {"name": self.name,
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
95 "key": escape(infoid)}))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
97 ''' def set_no_tranaction(self, infoid, **newvalues):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 """ this is missing transaction and may be affected by
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
99 a race condition on update. This will work for
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
100 redis-like embedded databases that don't support
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
101 watch/multi/exec
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
102 """
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103 infoid = self.makekey(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 timestamp=None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 values = self.redis.get(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 if values is not None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 values = self.todict(values)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
108 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
109 values={}
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
110 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
111 timestamp = float(values['__timestamp'])
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
112 except KeyError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
113 pass # stay at None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
114
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
115 if '__timestamp' in newvalues:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
116 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
117 float(newvalues['__timestamp'])
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
118 except ValueError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
119 # keep original timestamp if present
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
120 newvalues['__timestamp'] = timestamp or \
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
121 (time.time() + self.default_lifetime)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
122 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
123 newvalues['__timestamp'] = time.time() + self.default_lifetime
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
124
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
125 values.update(newvalues)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
126
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
127 self.redis.set(infoid, self.tostr(values))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
128 self.redis.expireat(infoid, int(values['__timestamp']))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
129 '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
130
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
131 def set(self, infoid, **newvalues):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
132 """ Implement set using watch/multi/exec to get some
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
133 protection against a change committing between
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
134 getting the data and setting new fields and
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
135 saving.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
136 """
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
137 infoid = self.makekey(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
138 timestamp = None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
139
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
140 with self.redis.pipeline() as transaction:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
141 # Give up and log after three tries.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
142 # Do not loop forever.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
143 for _retry in [1, 2, 3]:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
144 # I am ignoring transaction return values.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
145 # Assuming all errors will be via exceptions.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
146 # Not clear that return values that useful.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
147 transaction.watch(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
148 values = transaction.get(infoid)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
149 if values is not None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
150 values = self.todict(values)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
151 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
152 values = {}
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
153
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
154 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
155 timestamp = float(values['__timestamp'])
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
156 except KeyError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
157 pass # stay at None
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
158
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
159 if '__timestamp' in newvalues:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
160 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
161 float(newvalues['__timestamp'])
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
162 except ValueError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
163 # keep original timestamp if present
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
164 newvalues['__timestamp'] = timestamp or \
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
165 (time.time() + self.default_lifetime)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
166 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
167 newvalues['__timestamp'] = time.time() + \
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
168 self.default_lifetime
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
169
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
170 values.update(newvalues)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
171
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
172 transaction.multi()
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
173 transaction.set(infoid, self.tostr(values))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
174 transaction.expireat(infoid, int(values['__timestamp']))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
175 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
176 # assume this works or raises an WatchError
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
177 # exception indicating I need to retry.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
178 # Since this is not a transaction, an error
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
179 # in one step doesn't roll back other changes.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
180 # so I again ignore the return codes as it is not
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
181 # clear that I can do the rollback myself.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
182 # Format and other errors (e.g. expireat('d', 'd'))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
183 # raise exceptions tht bubble up and result in mail
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
184 # to admin.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
185 transaction.execute()
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
186 break
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
187 except redis.Exceptions.WatchError:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
188 logging.getLogger('roundup.redis').info(
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
189 _('Key %(key)s changed in %(name)s db' %
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
190 {"key": escape(infoid), "name": self.name})
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
191 )
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
192 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
193 raise Exception(_("Redis set failed afer 3 retries"))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
194
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
195 def list(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
196 return list(self.redis.keys(self.makekey('*')))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
197
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
198 def destroy(self, infoid=None):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
199 '''use unlink rather than delete as unlink is async and doesn't
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
200 wait for memory to be freed server-side
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
201 '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
202 self.redis.unlink(self.makekey(infoid))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
203
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
204 def commit(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
205 ''' no-op '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
206 pass
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
207
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
208 def lifetime(self, key_lifetime=None):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
209 """Return the proper timestamp to expire a key with key_lifetime
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
210 specified in seconds. Default lifetime is self.default_lifetime.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
211 """
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
212 return time.time() + (key_lifetime or self.default_lifetime)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
213
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
214 def updateTimestamp(self, sessid):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
215 ''' Other backends update only if timestamp would change by more
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
216 than 60 seconds. To do this in redis requires:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
217 get data _timestamp
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
218 calculate if update needed
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
219 if needed,
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
220 set new timestamp
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
221 why bother. Just set and forget.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
222 '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
223 # no need to do timestamp calculations
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
224 lifetime = self.lifetime()
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
225 # note set also updates the expireat on the key in redis
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
226 self.set(sessid, __timestamp=lifetime)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
227
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
228 def clean(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
229 ''' redis handles key expiration, so nothing to do here.
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
230 '''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
231 pass
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
232
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
233 def close(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
234 ''' redis uses a connection pool that self manages, so nothing
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
235 to do on close.'''
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
236 pass
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
237
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
238
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
239 class Sessions(BasicDatabase):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
240 name = 'sessions'
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
241
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
242
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
243 class OneTimeKeys(BasicDatabase):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
244 name = 'otks'
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
245
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
246 # vim: set sts ts=4 sw=4 et si :
Roundup Issue Tracker: http://roundup-tracker.org/