Mercurial > p > roundup > code

annotate roundup/rate_limit.py @ 6565:2c2dbfc332ba

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Try to handle multiple connections better. The session database is a hot spot. When multiple requests (e.g. 20) come in at the same time session database contention can get great. The original code didn't retry session database access when the open failed. This resulted in errors at the client. The second pass delayed 0.01 seconds and retried. It was better but we still had multiple second stalls. I think the first request got in, everybody else backed up and then retried at the same time. Again they stepped on each other. With logging I would see many counters go all the way to low single digits or to -1 indicating falure. This pass uses randomint to generate delays from 0-.125 seconds in 5ms increments. This performs better in testing. I rarely saw a counter less than 13 (2 failed retries). Current logging starts after 6 failures and counts down until success or failure.
author John Rouillard <rouilj@ieee.org>
date Thu, 16 Dec 2021 20:02:00 -0500
parents 69a35d164a69
children 8f29e4ea05ce
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 # Originaly from
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 # https://smarketshq.com/implementing-gcra-in-python-5df1f11aaa96?gi=4b9725f99bfa
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 # with imports, modifications for python 2, implementation of
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 # set/get_tat and marshaling as string, support for testonly
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 # and status method.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 from datetime import timedelta, datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
9
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
10 class RateLimit: # pylint: disable=too-few-public-methods
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 def __init__(self, count, period):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 self.count = count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 self.period = period
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 @property
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 def inverse(self):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 return self.period.total_seconds() / self.count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19
5722
2f116ba7e7cf Rename Store class in rate_limit.py to Gcra. The name Store makes no
John Rouillard <rouilj@ieee.org>
parents: 5717
diff changeset
20 class Gcra:
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
21 def __init__(self):
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
22 self.memory = {}
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 def get_tat(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 # This should return a previous tat for the key or the current time.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 return self.memory[key]
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 return datetime.min
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
30
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 def set_tat(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 self.memory[key] = tat
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 def get_tat_as_string(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35 # get value as string:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 # to allow it to be marshalled/unmarshaled
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 return self.memory[key].isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41 return datetime.min.isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 def set_tat_as_string(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 # Take value as string and unmarshall:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46 # to datetime
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
47 self.memory[key] = datetime.strptime(tat, "%Y-%m-%dT%H:%M:%S.%f")
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 def update(self, key, limit, testonly=False):
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
50 '''Determine if the item associated with the key should be
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 rejected given the RateLimit limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 tat = max(self.get_tat(key), now)
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
55 separation = (tat - now).total_seconds()
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 max_interval = limit.period.total_seconds() - limit.inverse
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 if separation > max_interval:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 reject = True
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 reject = False
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 if not testonly:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 new_tat = max(tat, now) + timedelta(seconds=limit.inverse)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 self.set_tat(key, new_tat)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 return reject
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66 def status(self, key, limit):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67 '''Return status suitable for displaying as headers:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 X-RateLimit-Limit: calls allowed per period. Period/window
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 is not specified in any api I found.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 X-RateLimit-Limit-Period: Non standard. Defines period in
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 seconds for RateLimit-Limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 X-RateLimit-Remaining: How many calls are left in this window.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 X-RateLimit-Reset: window ends in this many seconds (not an
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 epoch timestamp) and all RateLimit-Limit calls are
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 available again.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 Retry-After: if user's request fails, this is the next time there
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 will be at least 1 available call to be consumed.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 ret = {}
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 tat = self.get_tat(key)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 # static defined headers according to limit
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
83 # all values are strings as that is required when used as headers
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
84 ret['X-RateLimit-Limit'] = str(limit.count)
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
85 ret['X-RateLimit-Limit-Period'] = str(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
86 int(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
87 limit.period.total_seconds())
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
88 )
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 # status of current limit as of now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
93 current_count = int((limit.period - (tat - now)).total_seconds() /
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
94 limit.inverse)
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
95 ret['X-RateLimit-Remaining'] = str(min(current_count, limit.count))
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
97 # tat_in_epochsec = (tat - datetime(1970, 1, 1)).total_seconds()
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 seconds_to_tat = (tat - now).total_seconds()
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
99 ret['X-RateLimit-Reset'] = str(max(seconds_to_tat, 0))
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
100 ret['X-RateLimit-Reset-date'] = "%s" % tat
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
101 ret['Now'] = str((now - datetime(1970, 1, 1)).total_seconds())
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
102 ret['Now-date'] = "%s" % now
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 if self.update(key, limit, testonly=True):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 # A new request would be rejected if it was processes.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 # The user has to wait until an item is dequeued.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 # One item is dequeued every limit.inverse seconds.
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
108 ret['Retry-After'] = str(int(limit.inverse))
5996
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
109 ret['Retry-After-Timestamp'] = "%s" % \
69a35d164a69 Make rate_limit.py pass flake8.
John Rouillard <rouilj@ieee.org>
parents: 5937
diff changeset
110 (now + timedelta(seconds=limit.inverse)) # noqa: E127
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
111 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
112 # if we are not rejected, the user can post another
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
113 # attempt immediately.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
114 # Do we even need this header if not rejected?
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
115 # RFC implies this is used with a 503 (or presumably
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
116 # 429 which may postdate the rfc). So if no error, no header?
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
117 # ret['Retry-After'] = '0'
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
118 # ret['Retry-After-Timestamp'] = str(ret['Now-date'])
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
119 pass
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
120
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
121 return ret
Roundup Issue Tracker: http://roundup-tracker.org/