Commit 7a3724f3 authored by Srikanth Chelluri's avatar Srikanth Chelluri
Browse files

fix: handle empty 'Retry-After' header from GitLab 

When requests are throttled (HTTP response code 429), python-gitlab
assumed that 'Retry-After' existed in the response headers. This is
not always the case and so the request fails due to a KeyError. The
change in this commit adds a rudimentary exponential backoff to the
'http_request' method, which defaults to 10 retries but can be set
to -1 to retry without bound.
parent ce2c8356

docs/api-usage.rst

+15 −1
Original line number Diff line number Diff line
@@ -299,7 +299,9 @@ Rate limits 


python-gitlab obeys the rate limit of the GitLab server by default.  On

receiving a 429 response (Too Many Requests), python-gitlab sleeps for the

amount of time in the Retry-After header that GitLab sends back.

amount of time in the Retry-After header that GitLab sends back.  If GitLab

does not return a response with the Retry-After header, python-gitlab will

perform an exponential backoff.



If you don't want to wait, you can disable the rate-limiting feature, by

supplying the ``obey_rate_limit`` argument.
@@ -312,6 +314,18 @@ supplying the ``obey_rate_limit`` argument. 
   gl = gitlab.gitlab(url, token, api_version=4)

   gl.projects.list(all=True, obey_rate_limit=False)



If you do not disable the rate-limiting feature, you can supply a custom value

for ``max_retries``; by default, this is set to 10. To retry without bound when

throttled, you can set this parameter to -1. This parameter is ignored if

``obey_rate_limit`` is set to ``False``.



.. code-block:: python



   import gitlab

   import requests



   gl = gitlab.gitlab(url, token, api_version=4)

   gl.projects.list(all=True, max_retries=12)



.. warning::

gitlab/__init__.py

+11 −3
Original line number Diff line number Diff line
@@ -477,6 +477,10 @@ class Gitlab(object): 
        # obey the rate limit by default

        obey_rate_limit = kwargs.get("obey_rate_limit", True)



        # set max_retries to 10 by default, disable by setting it to -1

        max_retries = kwargs.get("max_retries", 10)

        cur_retries = 0



        while True:

            result = self.session.send(prepped, timeout=timeout, **settings)
@@ -486,7 +490,11 @@ class Gitlab(object): 
                return result



            if 429 == result.status_code and obey_rate_limit:

                if max_retries == -1 or cur_retries < max_retries:

                    wait_time = 2 ** cur_retries * 0.1

                    if "Retry-After" in result.headers:

                        wait_time = int(result.headers["Retry-After"])

                    cur_retries += 1

                    time.sleep(wait_time)

                    continue