Mercurial > p > roundup > code

annotate roundup/rate_limit.py @ 5937:5d0873a4de4a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix rate limit headers - were ints/floats need to be strings Running under gunicorn rest requests were crashing. Not all of the values for the rate limit headers were strings. Some were numbers. This caused the header generation for wsgi to fail. Now the values are all strings.
author John Rouillard <rouilj@ieee.org>
date Sun, 20 Oct 2019 20:56:56 -0400
parents e225f403cc35
children 69a35d164a69
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 # Originaly from
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 # https://smarketshq.com/implementing-gcra-in-python-5df1f11aaa96?gi=4b9725f99bfa
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 # with imports, modifications for python 2, implementation of
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 # set/get_tat and marshaling as string, support for testonly
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 # and status method.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 from datetime import timedelta, datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
9 class RateLimit: # pylint: disable=too-few-public-methods
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 def __init__(self, count, period):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 self.count = count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 self.period = period
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14 @property
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 def inverse(self):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 return self.period.total_seconds() / self.count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18
5722
2f116ba7e7cf Rename Store class in rate_limit.py to Gcra. The name Store makes no
John Rouillard <rouilj@ieee.org>
parents: 5717
diff changeset
19 class Gcra:
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
20 def __init__(self):
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
21 self.memory = {}
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 def get_tat(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # This should return a previous tat for the key or the current time.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 return self.memory[key]
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 return datetime.min
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 def set_tat(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 self.memory[key] = tat
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 def get_tat_as_string(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 # get value as string:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # to allow it to be marshalled/unmarshaled
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 return self.memory[key].isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 return datetime.min.isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 def set_tat_as_string(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 # Take value as string and unmarshall:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46 # to datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
47 self.memory[key] = datetime.strptime(tat,"%Y-%m-%dT%H:%M:%S.%f")
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 def update(self, key, limit, testonly=False):
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
50 '''Determine if the item associated with the key should be
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 rejected given the RateLimit limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 tat = max(self.get_tat(key), now)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 separation = (tat - now).total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 max_interval = limit.period.total_seconds() - limit.inverse
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 if separation > max_interval:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 reject = True
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 reject = False
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 if not testonly:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 new_tat = max(tat, now) + timedelta(seconds=limit.inverse)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 self.set_tat(key, new_tat)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 return reject
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66 def status(self, key, limit):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67 '''Return status suitable for displaying as headers:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 X-RateLimit-Limit: calls allowed per period. Period/window
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 is not specified in any api I found.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 X-RateLimit-Limit-Period: Non standard. Defines period in
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 seconds for RateLimit-Limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 X-RateLimit-Remaining: How many calls are left in this window.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 X-RateLimit-Reset: window ends in this many seconds (not an
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 epoch timestamp) and all RateLimit-Limit calls are
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 available again.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 Retry-After: if user's request fails, this is the next time there
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 will be at least 1 available call to be consumed.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 ret = {}
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 tat = self.get_tat(key)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 # static defined headers according to limit
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
83 # all values are strings as that is required when used as headers
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
84 ret['X-RateLimit-Limit'] = str(limit.count)
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
85 ret['X-RateLimit-Limit-Period'] = str(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
86 int(
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
87 limit.period.total_seconds())
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
88 )
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 # status of current limit as of now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
93 current_count = int((limit.period - (tat - now)).total_seconds()\
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
94 /limit.inverse)
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
95 ret['X-RateLimit-Remaining'] = str(min(current_count,limit.count))
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96
5739
e225f403cc35 Run pylint and clean up it's issues. Also fix comment.
John Rouillard <rouilj@ieee.org>
parents: 5722
diff changeset
97 # tat_in_epochsec = (tat - datetime(1970, 1, 1)).total_seconds()
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 seconds_to_tat = (tat - now).total_seconds()
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
99 ret['X-RateLimit-Reset'] = str(max(seconds_to_tat, 0))
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
100 ret['X-RateLimit-Reset-date'] = "%s"%tat
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
101 ret['Now'] = str((now - datetime(1970,1,1)).total_seconds())
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
102 ret['Now-date'] = "%s"%now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 if self.update(key, limit, testonly=True):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 # A new request would be rejected if it was processes.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 # The user has to wait until an item is dequeued.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 # One item is dequeued every limit.inverse seconds.
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
108 ret['Retry-After'] = str(int(limit.inverse))
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
109 ret['Retry-After-Timestamp'] = "%s"%(now + timedelta(seconds=limit.inverse))
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
110 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
111 # if we are not rejected, the user can post another
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
112 # attempt immediately.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
113 # Do we even need this header if not rejected?
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
114 # RFC implies this is used with a 503 (or presumably
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
115 # 429 which may postdate the rfc). So if no error, no header?
5937
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
116 # ret['Retry-After'] = '0'
5d0873a4de4a fix rate limit headers - were ints/floats need to be strings
John Rouillard <rouilj@ieee.org>
parents: 5739
diff changeset
117 # ret['Retry-After-Timestamp'] = str(ret['Now-date'])
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
118 pass
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
119
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
120 return ret
Roundup Issue Tracker: http://roundup-tracker.org/