Mercurial > p > roundup > code
view roundup/support.py @ 6814:3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
Allow admin to select the backend data store. Compatibility matrix:
main\/ session>| anydbm | sqlite | redis | mysql | postgresql |
anydbm | D | | X | | |
sqlite | X | D | X | | |
mysql | | | | D | |
postgresql | | | | | D |
--------------------------------------------------------------+
D - default if unconfigured, X - compatible choice
DETAILS
roundup/configuration.py:
add config.ini section sessiondb with settings: backend and redis_url.
CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt:
doc on config of session db and redis. Plus some other fixes:
admin - clarified why we do not drop __words and __testids
table in native-fts conversion. TYpo fix.
upgrading - doc how you can keep using anydbm for session data with
sqlite. Fix dupe sentence in an upgrading config.ini
section.
roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py:
code to support redis, redis/anydbm backends respectively.
roundup/backends/sessions_redis.py
new storage backend for redis.
roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py
redis uses a different way of calculating lifetime/timestamp.
Since expiration of an item occurred if its timestamp was more
than 1 week old, code would calculate:
now - 1 week + lifetime.
But this results in faster expiration in redis if used for
lifetime/timestamp.
Convert code to use the lifetime() method in BasicDatabase
that generates the right timestamp for each backend.
test/session_common.py:
added tests for more cases, get without default, getall non-existing
key etc. timestamp test changed to use new self.get_ts which is
overridden in other tests. Test that datatypes survive storage.
test/test_redis_session.py:
test redis session store with sqlite and anydbm primary databases
test/test_anydbm.py, test/test_sqlite.py
add test to make sure the databases are properly set up
sqlite - add test cases where anydbm is used as datastore
anydbm - remove updateTimestamp override add get_ts().
test/test_config.py
tests on redis_url and compatibility on choice of sessiondb backend
.travis.yml:
add redis db and redis-py
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 04 Aug 2022 14:41:58 -0400 |
| parents | 01643d37785f |
| children | 33eb82ad26ba |
line wrap: on
line source
"""Implements various support classes and functions used in a number of places in Roundup code. """ from __future__ import print_function __docformat__ = 'restructuredtext' import os, time, sys class TruthDict: '''Returns True for valid keys, False for others. ''' def __init__(self, keys): if keys: self.keys = {} for col in keys: self.keys[col] = 1 def __getitem__(self, name): if hasattr(self, 'keys'): return name in self.keys else: return True def ensureParentsExist(dest): if not os.path.exists(os.path.dirname(dest)): os.makedirs(os.path.dirname(dest)) class PrioList: '''Manages a sorted list. Currently only implements method 'append' and iteration from a full list interface. Implementation: We manage a "sorted" status and sort on demand. Appending to the list will require re-sorting before use. >>> p = PrioList() >>> for i in 5,7,1,-1: ... p.append(i) ... >>> for k in p: ... print k ... -1 1 5 7 ''' def __init__(self): self.list = [] self.sorted = True def append(self, item): self.list.append(item) self.sorted = False def __iter__(self): if not self.sorted: self.list.sort() self.sorted = True return iter(self.list) class Progress: '''Progress display for console applications. See __main__ block at end of file for sample usage. ''' def __init__(self, info, sequence): self.info = info self.sequence = iter(sequence) self.total = len(sequence) self.start = self.now = time.time() self.num = 0 self.stepsize = self.total // 100 or 1 self.steptimes = [] self.display() def __iter__(self): return self def __next__(self): self.num += 1 if self.num > self.total: print(self.info, 'done', ' '*(75-len(self.info)-6)) sys.stdout.flush() return next(self.sequence) if self.num % self.stepsize: return next(self.sequence) self.display() return next(self.sequence) # Python 2 compatibility: next = __next__ def display(self): # figure how long we've spent - guess how long to go now = time.time() steptime = now - self.now self.steptimes.insert(0, steptime) if len(self.steptimes) > 5: self.steptimes.pop() steptime = sum(self.steptimes) / len(self.steptimes) self.now = now eta = steptime * ((self.total - self.num)/self.stepsize) # tell it like it is (or might be) if now - self.start > 3: M = eta / 60 H = M / 60 M = M % 60 S = eta % 60 if self.total: s = '%s %2d%% (ETA %02d:%02d:%02d)' % (self.info, self.num * 100. / self.total, H, M, S) else: s = '%s 0%% (ETA %02d:%02d:%02d)' % (self.info, H, M, S) elif self.total: s = '%s %2d%%' % (self.info, self.num * 100. / self.total) else: s = '%s %d done' % (self.info, self.num) sys.stdout.write(s + ' '*(75-len(s)) + '\r') sys.stdout.flush() # vim: set et sts=4 sw=4 :
