Skip to content

fix(bigquery-jdbc): maxResults was not honored for child jobs - #14973

Open
logachev wants to merge 1 commit into
mainfrom
kirl/fix_script_readapi
Open

logachev wants to merge 1 commit into
mainfrom
kirl/fix_script_readapi

Conversation

@logachev

Copy link
Copy Markdown
Contributor

Summary

Multi-statement SCRIPT queries failed to activate the BigQuery Storage Read API (HTAPI / Arrow) on child SELECT jobs even when EnableHighThroughputAPI=1 and result sizes exceeded the Read API activation thresholds.

Root Cause

  • For standalone SELECT queries, BigQueryStatement.executeJob(...) fetches initial results via job.getQueryResults(QueryResultsOption.pageSize(querySettings.getMaxResultPerPage())) (default MaxResults=10000).
  • QueryJobConfiguration.Builder.setMaxResults(...) on a SCRIPT query only applies to the parent coordinator job, not to the child jobs spawned by BigQuery.
  • Both SCRIPT child-job result paths — BigQueryStatement.handleQueryResult(...) (case SCRIPT:) and BigQueryStatement.getMoreResultsImpl(...) — called currentJob.getQueryResults() with no options. Without a pageSize option, tabledata.list fetches up to 10 MB of rows in the first REST page, either fitting the entire result into page 1 (!results.hasNextPage()) or inflating rowsInPage so meetsReadRatio(...) evaluates to false and silently falls back to BigQueryJsonResultSet.

Fix

  • Added a BigQueryStatement.handleQueryResult(String query, Job job) overload that resolves StatementType and SqlType from job, fetches results with QueryResultsOption.pageSize(querySettings.getMaxResultPerPage()), and delegates to handleQueryResult(query, results, sqlType, job).
  • Updated both case SCRIPT: in handleQueryResult and getMoreResultsImpl to delegate to handleQueryResult(query, currentJob), eliminating duplicated child-job handling logic.

Tests

  • BigQueryStatementTest: added unit tests covering handleQueryResult(String, Job) directly as well as multi-statement SCRIPT execution across execute(...) and getMoreResults(), verifying QueryResultsOption.pageSize(maxResults) is applied to every child job and BigQueryArrowResultSet is activated for large SELECT child jobs while preserving DML update counts and BigQueryJsonResultSet for small SELECT child jobs.
  • ITStatementTest: updated testScript to assert BigQueryArrowResultSet and row counts across all 3 script statements via getMoreResults().
  • ITCallableStatementTest: added testCallableStatementScriptWithReadApi covering CALL + multi-SELECT script execution with Read API enabled.

@logachev
logachev requested review from a team as code owners November 11, 2026 19:45

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors result handling in BigQueryStatement to process query results directly from Job objects, enabling dynamic SQL type determination and consistent application of maximum results page size settings across child jobs during script executions. Additionally, new unit and integration tests have been added to verify script execution behavior and result set selection (JSON vs. Arrow/Read API). I have no feedback to provide as there are no review comments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant