Repository navigation
Handle pagination of results with more than 10,000 results gracefully #1506
Description
Description of the problem, including code/CLI snippet
Currently if you try to get a list with a lot of results, eg. the main GitLab project's issues list
gl = gitlab.Gitlab('https://gitlab.com', private_token='XXXXXXX')
gl.auth()
gitlab_prj = gl.projects.get(278964, lazy=True)
issues = gitlab_prj.issues.list(order_by='updated_at', sort='asc', per_page=100, as_list=False)You'll get an error when you try and use the list:
Exception has occurred: TypeError
int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Due to the fact that, for performance reasons GitLab don't return certain headers when the responses exceed 10,000: https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers
Expected Behavior
Gracefully handle this lack of x-total and x-total-pages header, either by limiting the results to 10,000 responses, or allowing a user to manually set the _list._total and _list._total_pages values.
Actual Behavior
Crashes with the above exception.
Specifications
- python-gitlab version:
2.7.1 - API version you are using (v3/v4):
v4 - Gitlab server version (or gitlab.com):
gitlab.com
Activity
JohnVillalovos commentedon Jun 10, 2021
I tried the above code and I didn't have an exception occur. I iterated to about 16,000+ issues before I broke out of the program.
VenkatsQuest commentedon Nov 10, 2021
This is the call stack , hope this information is helpful to fix the issue,
/Users/anil/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/gitlab/client.py
/Users/anil/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/gitlab/base.py
/Users/anil/PycharmProjects/pythonProject/main.py
iterator_projects = gl.projects.list(owned=True, as_list=False , pagination="keyset", order_by="id")
nejch commentedon Jan 4, 2022
Reproduced in #1788 and fixed in #1773.