Mercurial > p > roundup > code

annotate roundup/rate_limit.py @ 7399:deb8e7e6d66d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Skip redis tests if unable to communicate with the server. If the redis module is in the test environment, the redis tests will not be skipped. If connecting to redis during testing fails with a ConnectionError because there is no redis server at localhost, or if it fails with an AuthenticationError, you would fail a slew of tests. This causes the tests to report as skipped if either of the two errors occurs. It is very inefficient as it fails in setup() for the tests, but at least it does report skipping the tests. Also documented how to pass the redis password to the tests in the test part of the install docs. Future note: running tests needs proper docs in development.txt (including database setup) and a link left to that doc in installation.txt.
author John Rouillard <rouilj@ieee.org>
date Wed, 24 May 2023 12:52:53 -0400
parents 69a35d164a69
children 8f29e4ea05ce
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 # Originaly from
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 # https://smarketshq.com/implementing-gcra-in-python-5df1f11aaa96?gi=4b9725f99bfa
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 # with imports, modifications for python 2, implementation of
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 # set/get_tat and marshaling as string, support for testonly
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 # and status method.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 from datetime import timedelta, datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
9
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
10 class RateLimit: # pylint: disable=too-few-public-methods
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 def __init__(self, count, period):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 self.count = count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 self.period = period
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 @property
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 def inverse(self):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 return self.period.total_seconds() / self.count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19
5722
2f116ba7e7cf Rename Store class in rate_limit.py to Gcra. The name Store makes no
John Rouillard <rouilj@ieee.org>
parents: 5717
diff changeset
20 class Gcra:
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
21 def __init__(self):
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
22 self.memory = {}
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 def get_tat(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 # This should return a previous tat for the key or the current time.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 return self.memory[key]
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 return datetime.min
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
30
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 def set_tat(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 self.memory[key] = tat
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 def get_tat_as_string(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35 # get value as string:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 # to allow it to be marshalled/unmarshaled
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 return self.memory[key].isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41 return datetime.min.isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 def set_tat_as_string(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 # Take value as string and unmarshall:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46 # to datetime
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
47 self.memory[key] = datetime.strptime(tat, "%Y-%m-%dT%H:%M:%S.%f")
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 def update(self, key, limit, testonly=False):
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
50 '''Determine if the item associated with the key should be
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 rejected given the RateLimit limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 tat = max(self.get_tat(key), now)
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
55 separation = (tat - now).total_seconds()
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 max_interval = limit.period.total_seconds() - limit.inverse
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 if separation > max_interval:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 reject = True
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 reject = False
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 if not testonly:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 new_tat = max(tat, now) + timedelta(seconds=limit.inverse)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 self.set_tat(key, new_tat)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 return reject
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66 def status(self, key, limit):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67 '''Return status suitable for displaying as headers:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 X-RateLimit-Limit: calls allowed per period. Period/window
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 is not specified in any api I found.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 X-RateLimit-Limit-Period: Non standard. Defines period in
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 seconds for RateLimit-Limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 X-RateLimit-Remaining: How many calls are left in this window.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 X-RateLimit-Reset: window ends in this many seconds (not an
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 epoch timestamp) and all RateLimit-Limit calls are
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 available again.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 Retry-After: if user's request fails, this is the next time there
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 will be at least 1 available call to be consumed.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 ret = {}
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 tat = self.get_tat(key)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 # static defined headers according to limit
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
83 # all values are strings as that is required when used as headers
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
84 ret['X-RateLimit-Limit'] = str(limit.count)
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
85 ret['X-RateLimit-Limit-Period'] = str(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
86 int(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
87 limit.period.total_seconds())
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
88 )
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 # status of current limit as of now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
93 current_count = int((limit.period - (tat - now)).total_seconds() /
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
94 limit.inverse)
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
95 ret['X-RateLimit-Remaining'] = str(min(current_count, limit.count))
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
97 # tat_in_epochsec = (tat - datetime(1970, 1, 1)).total_seconds()
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 seconds_to_tat = (tat - now).total_seconds()
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
99 ret['X-RateLimit-Reset'] = str(max(seconds_to_tat, 0))
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
100 ret['X-RateLimit-Reset-date'] = "%s" % tat
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
101 ret['Now'] = str((now - datetime(1970, 1, 1)).total_seconds())
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
102 ret['Now-date'] = "%s" % now
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 if self.update(key, limit, testonly=True):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 # A new request would be rejected if it was processes.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 # The user has to wait until an item is dequeued.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 # One item is dequeued every limit.inverse seconds.
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
108 ret['Retry-After'] = str(int(limit.inverse))
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
109 ret['Retry-After-Timestamp'] = "%s" % \
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
110 (now + timedelta(seconds=limit.inverse)) # noqa: E127
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
111 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
112 # if we are not rejected, the user can post another
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
113 # attempt immediately.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
114 # Do we even need this header if not rejected?
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
115 # RFC implies this is used with a 503 (or presumably
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
116 # 429 which may postdate the rfc). So if no error, no header?
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
117 # ret['Retry-After'] = '0'
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
118 # ret['Retry-After-Timestamp'] = str(ret['Now-date'])
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
119 pass
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
120
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
121 return ret
Roundup Issue Tracker: http://roundup-tracker.org/