-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Closed
Closed
Copy link
Labels
api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.api: bigquerystorageIssues related to the BigQuery Storage API.Issues related to the BigQuery Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.
Description
Follow-up to #7552.
We now support displaying a progress bar in to_dataframe when the tabledata.list API is used, but not when the bqstorage_client is used. The BigQuery Storage API is faster, but not instantaneous, so it would still be useful to display a progress bar.
There are a couple of complications with this.
- When the BigQuery Storage API is used, a threadpool is used to download query results. The progress bar should only be updated from a single thread, so the threadsafe queue should probably be used from the worker threads to send updates to the main thread that updates the progress bar.
to_dataframein the BigQuery Storage API client only supports downloading a whole stream at once. The reader needs to be updated to provide apagesproperty to allow converting individual blocks to a pandas DataFrame. That way incremental updates can be sent.
Metadata
Metadata
Assignees
Labels
api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.api: bigquerystorageIssues related to the BigQuery Storage API.Issues related to the BigQuery Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.