Skip to content

Datastore: broken paging next_page_token #4347

@mathewcohle

Description

@mathewcohle

I want to use datastore paging to fetch the query results. Specifically, I upload 11 entities to datastore and then want to fetch them (limit=10) using following code:

query = <google.cloud.datastore.Client>.query(kind='test_kind')

query_iter = query.fetch(limit=limit, start_cursor=cursor)
page = next(query_iter.pages)
next_cursor = query_iter.next_page_token
entities = list(page)

logging.info('Cursor: %s, Next cursor: %s, Loaded entities: %d.',
              cursor, next_cursor, len(entities))

(where initially cursor = None and then is set to next_cursor value)

Unfortunately, I am getting not None value for the next_cursor even there are not any more results to fetch:

# First call, everything as expected
Cursor: None, Next cursor: foo, Loaded entities: 10.

# Second call, expected next_page_token = None
Cursor: foo, Next cursor: bar, Loaded entities: 1.

# Third call, again expected next_page_token = None
Cursor: bar, Next cursor: bar, Loaded entities: 0.

It seems that next_page_token is always not None and moreover, if there is not any result to fetch, next_cursor == cursor.

Setup:

Python 2.7.12
google-cloud-datastore==1.3.0

Is this expected? If yes, how can I recognise that there are not more results to fetch?

I've checked the code briefly and it might be related to https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/datastore/google/cloud/datastore/query.py#L457. It seems like response_pb.batch.end_cursor == b'' does not hold and thus next_page_token is not set to None, but not sure why this is the case.

Metadata

Metadata

Assignees

Labels

api: datastoreIssues related to the Datastore API.type: questionRequest for information or clarification. Not an issue.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions