Skip to content

BigQuery: Implement progress_bar_type support when bqstorage_client is used #7654

@tswast

Description

@tswast

Follow-up to #7552.

We now support displaying a progress bar in to_dataframe when the tabledata.list API is used, but not when the bqstorage_client is used. The BigQuery Storage API is faster, but not instantaneous, so it would still be useful to display a progress bar.

There are a couple of complications with this.

  • When the BigQuery Storage API is used, a threadpool is used to download query results. The progress bar should only be updated from a single thread, so the threadsafe queue should probably be used from the worker threads to send updates to the main thread that updates the progress bar.
  • to_dataframe in the BigQuery Storage API client only supports downloading a whole stream at once. The reader needs to be updated to provide a pages property to allow converting individual blocks to a pandas DataFrame. That way incremental updates can be sent.

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the BigQuery API.api: bigquerystorageIssues related to the BigQuery Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions