Default value for max_retries is too short to "clear" all rate limit windows

Description of the problem, including code/CLI snippet

The default value of max_retries is insufficient to "clear" any rate limiting limit, for responses where the Retry-After header is missing. The default is 10, which yields a max cur_retries value of 9, which yields a maximum sleep time of ~50 seconds. At least some rate limits are computed over a 10 minute window (600 seconds) (e.g. the recently added limit on /users/:id), so ideally we'd have a default max_retries that would (cumulatively) sleep at least that long.

Expected Behavior

Default max_retries value is large enough to cause the library to sleep long enough to pass any rate limit window.

Actual Behavior

The library doesn't sleep long enough, so a exception is thrown when the rate limit is reached, despite retrying for a while.

Specifications

python-gitlab version: 1.15.0 (but I checked the latest code in git and it appears identical)
API version you are using (v3/v4): Whatever the library default is.
Gitlab server version (or gitlab.com): gitlab.com

@swarren

Thanks for the report @swarren. Do you have a few links for some of these new endpoints and rate limit behaviors, maybe some upstream MR/issues for us to look at? :) Thanks. 10 minutes seems like quite a lot so we should maybe find some sensible approach, I like your idea in the other issue with global options.

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73069/diffs#1246552930c7d15d9888238d19f23329f673f4d6_52_52

BTW, I wrote a script to continually hit this endpoint, and found that the Retry-After header isn't sent for it:-(

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73069/diffs#1246552930c7d15d9888238d19f23329f673f4d6_52_52

BTW, I wrote a script to continually hit this endpoint, and found that the Retry-After header isn't sent for it:-(

Interesting, I guess GitLab is almost approaching something similar to GitHub's way of doing secondary rate limits and we'll have issues similar to PyGithub/PyGithub#2113. 🤔

I notice this on the user endpoint

I removed the non RateLimit headers.

{'RateLimit-Limit': '2000',
 'RateLimit-Observed': '20',
 'RateLimit-Remaining': '1980',
 'RateLimit-Reset': '1645057248',
 'RateLimit-ResetTime': 'Thu, 17 Feb 2022 00:20:48 GMT'}

Looks like we need to add support for these new headers: https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#response-headers

fix: support RateLimit-Reset header

@JohnVillalovos

@JohnVillalovos I don't believe that change will solve my original issue. There are no rate-limit or retry headers at all when the rate limit is hit on /users/:id:

for i in $(seq 305); do curl --verbose --header "Authorization: Bearer xxx" "https://gitlab.com/api/v4/users/5400102"; done 2>&1 | tee log.txt

eventually yields:

HTTP/2 429 
< date: Thu, 10 Mar 2022 04:08:28 GMT
< content-type: application/json
< content-length: 89
< cache-control: no-cache
< vary: Origin
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-request-id: 01FXS0728WE6KDCRW14TR5RE9P
< x-runtime: 0.026642
< strict-transport-security: max-age=31536000
< referrer-policy: strict-origin-when-cross-origin
< gitlab-lb: fe-01-lb-gprd
< gitlab-sv: api-gke-us-east1-c
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=JfaflNapv18Po%2BZPk13XOp2%2FJJE0oSpTsz4CtpzSA3f0RBuphqWYa%2BWImkWAzxE5Urk20dQtug9veDTFkRFPqyCkK6WOr9Lzk7tTT7XbqrhnZBaZMFuj4pkKYcpaE7mhuySC8V%2BkDMI%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6e99307aeccfc7b5-DEN
< 
{ [89 bytes data]
100    89  100    89    0     0    380      0 --:--:-- --:--:-- --:--:--   378

@JohnVillalovos

@JohnVillalovos I don't believe that change will solve my original issue. There are no rate-limit or retry headers at all when the rate limit is hit on /users/:id:

Has an issue been filed with GitLab to figure out if this is "as intended" or an oversight on their part?

I haven't filed a bug

Gitlab support did notify me that the server-side changes would affect my usage, and I mentioned to them that python-gitlab should handle this based on rate limit headers so I wouldn't need to change anything. Then when the server-side change was rolled out I notified them again that python-gitlab wasn't able to handle this due to this missing rate limit headers. They didn't reply to that last point. Thus, at least part of gitlab should be aware of this, although I haven't formally filed a bug.

Hello, I've also encountered this issue while working with gitlab.com API. It seems that the "Retry-After", etc. headers are missing, hence the default wait_time set by python-gitlab is too low.
IMO we should:

Set the default wait time should be 1 minute since gitlab.com rate limits are calculate per 1 minute.
Open a bug with gitlab's team.

edit: opened a bug request with them
https://gitlab.com/gitlab-org/gitlab/-/issues/365728

This issue was marked stale because it has been open 60 days with
no activity. Please remove the stale label or comment on this
issue. Otherwise, it will be closed in 15 days.

As an open-source project, we rely on community contributions to
address many of the reported issues. Without a proposed fix or
active work towards a solution it is our policy to close inactive
issues. This is documented in CONTRIBUTING.rst

How to keep this issue open:

If you are still experiencing this issue and are willing to
investigate a fix, please comment and let us know.
If you (or someone else) can propose a pull request with a
solution, that would be fantastic.
Any significant update or active discussion indicating progress
will also prevent closure.

We value your input. If you can help provide a fix, we'd be happy
to keep this issue open and support your efforts.

This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst

This issue was closed because it has been marked stale for 15 days
with no activity.

This open-source project relies on community contributions, and
while we value all feedback, we have a limited capacity to address
every issue without a clear path forward.

Currently, this issue hasn't received a proposed fix, and there
hasn't been recent active discussion indicating someone is planning
to work on it. To maintain a manageable backlog and focus our
efforts, we will be closing this issue for now.

This doesn't mean the issue isn't valid or important. If you or
anyone else in the community is willing to investigate and propose
a solution (e.g., by submitting a pull request), please do.

We believe that those who feel a bug is important enough to fix
should ideally be part of the solution. Your contributions are
highly welcome.

Thank you for your understanding and potential future
contributions.

This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst

Navigation Menu

Repository navigation

Default value for max_retries is too short to "clear" all rate limit windows #1889

Description of the problem, including code/CLI snippet

Expected Behavior

Actual Behavior

Specifications

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Default value for max_retries is too short to "clear" all rate limit windows #1889

Description

Description of the problem, including code/CLI snippet

Expected Behavior

Actual Behavior

Specifications

Activity

nejch commented on Feb 10, 2022

swarren commented on Feb 10, 2022

nejch commented on Feb 10, 2022

JohnVillalovos commented on Feb 17, 2022

JohnVillalovos commented on Feb 17, 2022

swarren commented on Mar 10, 2022

JohnVillalovos commented on Mar 10, 2022

swarren commented on Mar 10, 2022

bachsh commented on Jun 20, 2022

github-actions commented on Jul 9, 2025

github-actions commented on Jul 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions