Skip to content

feat(bigquery): accelerate row-based query() with Arrow wire format #14355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jinseopkim0 wants to merge 6 commits into from
Closed

feat(bigquery): accelerate row-based query() with Arrow wire format #14355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
97b8f78
feat(bigquery): add zero-copy queryArrow API for Arrow VectorSchemaRo…
jinseopkim0 Sep 14, 2026
ffef9ff
fix(bigquery): address bot review comments on queryArrow and settings
jinseopkim0 Sep 14, 2026
ebc1c6c
fix(bigquery): support IPv6 loopback for plaintext channel configuration
jinseopkim0 Sep 14, 2026
4d1e4ee
test(bigquery): add ITBigQueryTest integration tests for queryArrow
jinseopkim0 Sep 14, 2026
21d547f
feat(bigquery): add ArrowQueryPageFetcher for Arrow query result pagi…
jinseopkim0 Sep 14, 2026
be77a7d
feat(bigquery): accelerate row-based query() with Arrow wire format
jinseopkim0 Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions java-bigquery/google-cloud-bigquery/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,15 @@
<artifactId>arrow-memory-netty</artifactId>
</dependency>

<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax-grpc</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-api</artifactId>
</dependency>

<dependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
Expand Down
52 changes: 52 additions & 0 deletions java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1639,6 +1639,58 @@ TableResult query(QueryJobConfiguration configuration, JobOption... options)
TableResult query(QueryJobConfiguration configuration, JobId jobId, JobOption... options)
throws InterruptedException, JobException;

/**
* <b>[Beta]</b> Runs the query associated with the request and returns an {@link
* ArrowQueryResult} yielding Apache Arrow {@code VectorSchemaRoot} batches directly for zero-copy
* vector access.
*
* <p>Callers must manage off-heap native memory by closing the returned {@link ArrowQueryResult}
* (e.g. via a {@code try-with-resources} block).
*
* <p><b>Prerequisite:</b> Requires the BigQuery Storage Read API ({@code
* bigquerystorage.googleapis.com}) to be enabled on your GCP project.
*
* @param configuration the query configuration
* @param options query options
* @return an {@link ArrowQueryResult} streaming Arrow vectors
* @throws BigQueryException upon failure
* @throws InterruptedException if the current thread gets interrupted while waiting for the query
* to complete
* @throws JobException if the job completes unsuccessfully
*/
@BetaApi
default ArrowQueryResult queryArrow(QueryJobConfiguration configuration, JobOption... options)
throws InterruptedException, JobException {
throw new UnsupportedOperationException("queryArrow is not implemented");
}

/**
* <b>[Beta]</b> Runs the query associated with the request, using the given JobId, and returns an
* {@link ArrowQueryResult} yielding Apache Arrow {@code VectorSchemaRoot} batches directly for
* zero-copy vector access.
*
* <p>Callers must manage off-heap native memory by closing the returned {@link ArrowQueryResult}
* (e.g. via a {@code try-with-resources} block).
*
* <p><b>Prerequisite:</b> Requires the BigQuery Storage Read API ({@code
* bigquerystorage.googleapis.com}) to be enabled on your GCP project.
*
* @param configuration the query configuration
* @param jobId the job ID to use
* @param options query options
* @return an {@link ArrowQueryResult} streaming Arrow vectors
* @throws BigQueryException upon failure
* @throws InterruptedException if the current thread gets interrupted while waiting for the query
* to complete
* @throws JobException if the job completes unsuccessfully
*/
@BetaApi
default ArrowQueryResult queryArrow(
QueryJobConfiguration configuration, JobId jobId, JobOption... options)
throws InterruptedException, JobException {
throw new UnsupportedOperationException("queryArrow is not implemented");
}

/**
* Starts the query associated with the request, using the given JobId. It returns either
* TableResult for quick queries or Job object for long-running queries.
Expand Down
Loading
Loading