Mercurial > p > roundup > code

annotate roundup/rate_limit.py @ 5717:cad18de2b988

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
issue2550949: Rate limit password guesses/login attempts. Generic rate limit mechanism added. Deployed for web page logins. Default is 3 login attempts/minute for a user. After which one login attempt every 20 seconds can be done. Uses gcra algorithm so all I need to store is a username and timestamp in the one time key database. This does mean I don't have a list of all failed login attempts as part of the rate limiter. Set up config setting as well so admin can tune the rate. Maybe 1 every 10 seconds is ok at a site with poor typists who need 6 attempts to get the password right 8-). The gcra method can also be used to limit the rest and xmlrpc interfaces if needed. The mechanism I added also supplies a status method that calculates the expected values for http headers returned as part of rate limiting. Also tests added to test all code paths I hope.
author John Rouillard <rouilj@ieee.org>
date Sat, 11 May 2019 17:24:58 -0400
parents
children 2f116ba7e7cf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5717
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 # Originaly from
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 # https://smarketshq.com/implementing-gcra-in-python-5df1f11aaa96?gi=4b9725f99bfa
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 # with imports, modifications for python 2, implementation of
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 # set/get_tat and marshaling as string, support for testonly
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 # and status method.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 from datetime import timedelta, datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 class RateLimit:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 def __init__(self, count, period):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 self.count = count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 self.period = period
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14 @property
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 def inverse(self):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 return self.period.total_seconds() / self.count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19 class Store:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 memory = {}
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 def get_tat(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # This should return a previous tat for the key or the current time.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 return self.memory[key]
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 return datetime.min
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 def set_tat(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 self.memory[key] = tat
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 def get_tat_as_string(self, key):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 # get value as string:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # to allow it to be marshalled/unmarshaled
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 if key in self.memory:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 return self.memory[key].isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 return datetime.min.isoformat()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 def set_tat_as_string(self, key, tat):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 # Take value as string and unmarshall:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 # YYYY-MM-DDTHH:MM:SS.mmmmmm
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46 # to datetime
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
47 self.memory[key] = datetime.strptime(tat,"%Y-%m-%dT%H:%M:%S.%f")
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 def update(self, key, limit, testonly=False):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
50 '''Determine if the item assocaited with the key should be
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 rejected given the RateLimit limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 tat = max(self.get_tat(key), now)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 separation = (tat - now).total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 max_interval = limit.period.total_seconds() - limit.inverse
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 if separation > max_interval:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 reject = True
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 reject = False
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 if not testonly:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 new_tat = max(tat, now) + timedelta(seconds=limit.inverse)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 self.set_tat(key, new_tat)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 return reject
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66 def status(self, key, limit):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67 '''Return status suitable for displaying as headers:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 X-RateLimit-Limit: calls allowed per period. Period/window
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 is not specified in any api I found.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 X-RateLimit-Limit-Period: Non standard. Defines period in
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 seconds for RateLimit-Limit.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 X-RateLimit-Remaining: How many calls are left in this window.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 X-RateLimit-Reset: window ends in this many seconds (not an
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 epoch timestamp) and all RateLimit-Limit calls are
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 available again.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 Retry-After: if user's request fails, this is the next time there
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 will be at least 1 available call to be consumed.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 '''
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 ret = {}
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 tat = self.get_tat(key)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
83 # static defined headers according to limit
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
84 ret['X-RateLimit-Limit'] = limit.count
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
85 ret['X-RateLimit-Limit-Period'] = limit.period.total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
86
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
87 # status of current limit as of now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
88 now = datetime.utcnow()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 ret['X-RateLimit-Remaining'] = min(int(
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 (limit.period - (tat - now)).total_seconds() \
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92 /limit.inverse),ret['X-RateLimit-Limit'])
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
93
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
94 tat_epochsec = (tat - datetime(1970, 1, 1)).total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
95 seconds_to_tat = (tat - now).total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96 ret['X-RateLimit-Reset'] = max(seconds_to_tat, 0)
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
97 ret['X-RateLimit-Reset-date'] = "%s"%tat
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 ret['Now'] = (now - datetime(1970,1,1)).total_seconds()
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
99 ret['Now-date'] = "%s"%now
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
100
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
101 if self.update(key, limit, testonly=True):
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
102 # A new request would be rejected if it was processes.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103 # The user has to wait until an item is dequeued.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 # One item is dequeued every limit.inverse seconds.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 ret['Retry-After'] = limit.inverse
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 ret['Retry-After-Timestamp'] = "%s"%(now + timedelta(seconds=limit.inverse))
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 else:
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
108 # if we are not rejected, the user can post another
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
109 # attempt immediately.
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
110 # Do we even need this header if not rejected?
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
111 # RFC implies this is used with a 503 (or presumably
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
112 # 429 which may postdate the rfc). So if no error, no header?
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
113 # ret['Retry-After'] = 0
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
114 # ret['Retry-After-Timestamp'] = ret['Now-date']
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
115 pass
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
116
cad18de2b988 issue2550949: Rate limit password guesses/login attempts.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
117 return ret
Roundup Issue Tracker: http://roundup-tracker.org/