Repository navigation
Rate limiting & recovery #166
Description
Would it be possible to add rate limiting into GitLab._raw_get (https://github.com/gpocentek/python-gitlab/blob/5f444e4ddf1087a8c4081f9b8d8b3d87d36a0985/gitlab/__init__.py), and tie it to a config option in ~/.python-gitlab.cfg?
I keep getting gitlab.exceptions.GitlabListError from HTTP 429 caused by gitlab rate limiting. I expect that I can catch this exception, delay the code, then retry the failed command. There is no nice way though to know the number of underlying API requests made by python-gitlab, so I cannot precisely figure out the delay, nor easily pre-delay to avoid the HTTP 429. I'm using a few ....list(all=True), so even with arbitrary sleeps here and there, I still hit the limit a lot.
Reacted by Federico Ceratto
Activity
gpocentek commentedon Oct 17, 2016
@jflemer-ndp did you try to pass a per_page argument to get more than 20 items per API request? I guess you have a lot of data to handle if you reach the limits, but maybe getting larger batch of data at a time would help. Something like
....list(all=True, per_page=100)If possible I'd like to avoid implementing an API limitation workaround.
Let me know if this works for you.
gpocentek commentedon Oct 17, 2016
Note: my company gitlab seems to limit the number of items per page to 100. Not sure if this is configurable on the server side.
jflemer-ndp commentedon Oct 17, 2016
With
per_page=100I get a little further, but still eventually hit the limit. I am looping over all the notes of all the tickets. Since there is no way in the GitLab API to filter what notes I want, this is going to be a lot of objects.gpocentek commentedon Oct 17, 2016
It was worth a try... I'll see what I can do but this might take me some time (bit swamped at the moment).
jflemer-ndp commentedon Oct 17, 2016
I happened to find a kinda cool idea about using a decorator for rate limiting here http://blog.gregburek.com/2011/12/05/Rate-limiting-with-decorators/
But that probably wouldn't fit with making it configurable.
If I get fed up with my excessive and arbitrary
sleeps, I'll try to submit a patch/merge request on this.max-wittig commentedon Apr 10, 2018
@jflemer-ndp Any progress on your implementation of rate limits, yet?
Maybe this library could be used? https://github.com/tomasbasham/ratelimit
feat: obey the rate limit
feat: obey the rate limit
feat: obey the rate limit