Repository navigation
Regression on calling len() on GitlabList objects since switching to keyset pagination by default #1114
Description
Description of the problem, including code/CLI snippet
Pull request #1110 that has been released in v2.3.0 yesterday introduced a regression on a specific scenario:
when calling len() on the output of Gitlab().projects.list(as_list=False).
Expected Behavior
Being able to call len() on the iterator returned by Gitlab().projects.list(as_list=False)
Actual Behavior
Minimal reproduction code:
import gitlab
len(gitlab.Gitlab(...).projects.list(as_list=False))Output:
Traceback (most recent call last):
File "test.py", line 3, in <module>
len(gitlab.Gitlab(...).projects.list(as_list=False))#, pagination='offset'))
File ".../lib/python3.7/site-packages/gitlab/base.py", line 184, in __len__
return len(self._list)
File ".../lib/python3.7/site-packages/gitlab/__init__.py", line 863, in __len__
return int(self._total)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Workaround:
import gitlab
len(gitlab.Gitlab(...).projects.list(as_list=False, pagination='offset'))Specifications
- python-gitlab version: sicnce v2.3.0
- API version you are using (v3/v4): v4
- Gitlab server version (or gitlab.com): 12.10.7-ee (80b99e43cfc)
Activity
Lucas-C commentedon Jun 9, 2020
#1110 introduced this code change to reflect the new pagination API of Gitlab : https://docs.gitlab.com/ce/api/README.html#keyset-based-pagination
But maybe this should not be the new value by default ?
max-wittig commentedon Jun 9, 2020
Thanks for the report. I guess this is yet another testcase for the functional tests.
Gitlab doesn't seem to include headers for keyset pagination☹️
max-wittig commentedon Jun 9, 2020
Offset pagination doesn't work, if you have over 50.000 projects and it's recommended by GitLab.
I will look at this upstream and revert the default for right now: https://gitlab.com/gitlab-org/gitlab/-/issues/220903
max-wittig commentedon Jun 9, 2020
@Lucas-C Reverted.
max-wittig commentedon Jun 16, 2020
@Lucas-C Sadly GitLab doesn't want to continue offering the total items in keyset pagination.
See: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34273#note_361863716. I would suggest to the just follow GitLab's default pagination (with sadly is still offset) and make it easier to change to
keyset(by automatically setting order_by='id' and only using keyset when in project context)What do you think? @bufferoverflow
Lucas-C commentedon Jun 16, 2020
I think there is a small typo here, as I don't quite understand :)
I'm fine with this solution.
Overall, I'd be very happy with any solution that is backward compatible, meaning being able to call
len()on.projects.list(as_list=False)output. Maybe a test for that scenario should be added by the way.max-wittig commentedon Jun 16, 2020
I've added it already 😉 e71fe16#diff-4a8817e452ed84e4d3b53c2523dac89aR424
Lucas-C commentedon Jun 16, 2020
Wonderful, thanks !
nejch commentedon Jan 4, 2022
This was fixed in #1773. The
len()won't return anything useful (0) - but we don't have that info from the GitLab server anyway.Feel free to reopen if it's still an issue after 3.0.0 :)