Skip to content

Handle pagination of results with more than 10,000 results gracefully #1506

@Siyfion

Description

@Siyfion

Description of the problem, including code/CLI snippet

Currently if you try to get a list with a lot of results, eg. the main GitLab project's issues list

gl = gitlab.Gitlab('https://gitlab.com', private_token='XXXXXXX')
gl.auth()
gitlab_prj = gl.projects.get(278964, lazy=True)
issues = gitlab_prj.issues.list(order_by='updated_at', sort='asc', per_page=100, as_list=False)

You'll get an error when you try and use the list:

Exception has occurred: TypeError
int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Due to the fact that, for performance reasons GitLab don't return certain headers when the responses exceed 10,000: https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Expected Behavior

Gracefully handle this lack of x-total and x-total-pages header, either by limiting the results to 10,000 responses, or allowing a user to manually set the _list._total and _list._total_pages values.

Actual Behavior

Crashes with the above exception.

Specifications

  • python-gitlab version: 2.7.1
  • API version you are using (v3/v4): v4
  • Gitlab server version (or gitlab.com): gitlab.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions