fix(bigquery-jdbc): scope Storage Read/Write API credentials for TPC - #14972

Summary

Queries routed through the BigQuery Storage Read API (HTAPI / Arrow) or Storage Write API fail with UNAUTHENTICATED when connecting to non-default universe domains (TPC / custom universeDomain) using service account or Application Default Credentials.

Root Cause

1. BigQueryJdbcOAuthUtility.getCredentials(...) returns unscoped GoogleCredentials (createScopedRequired() == true) unless RequestGoogleDriveScope=1 or service account impersonation is configured.
2. For the REST BigQuery client, BigQueryOptions (ServiceOptions.getScopedCredentials()) automatically scopes credentials, causing ServiceAccountCredentials in non-default universes to mint a scope-based self-signed JWT (scope claim, no aud claim).
3. For BigQueryReadClient and BigQueryWriteClient, BigQueryConnection bypassed BigQueryReadSettings.defaultCredentialsProviderBuilder() / BigQueryWriteSettings.defaultCredentialsProviderBuilder() and wrapped the unscoped credential directly in FixedCredentialsProvider.create(...). Because createScopedRequired() remained true, ServiceAccountCredentials fell back to minting an audience-based self-signed JWT (aud set to the custom universe Storage endpoint, no scope claim), which is rejected with UNAUTHENTICATED.

Fix

• Added BigQueryConnection.scopeStorageCredentials(Credentials, List<String>) mirroring GoogleCredentialsProvider in BigQueryReadSettings and BigQueryWriteSettings:
  • Applies default service scopes (BigQueryReadSettings.getDefaultServiceScopes() / BigQueryWriteSettings.getDefaultServiceScopes()) when createScopedRequired() is true, while preserving explicit scopes when already configured (such as RequestGoogleDriveScope=1).
  • Enables createWithUseJwtAccessWithScope(true) on ServiceAccountCredentials so self-signed JWTs include the scope claim without requiring extra OAuth token endpoint round-trips in GDU.
• Re-enabled Read API integration tests in TPC (@Tag("disable_tpc")) that were previously disabled due to this authentication failure.

Tests

• BigQueryConnectionTest: verifies BigQueryReadClient and BigQueryWriteClient preserve custom universeDomain settings and mint scope-based self-signed JWTs (scope claim present, aud absent), including when explicit scopes are pre-applied.
• ITBigQueryJDBCTest: re-enabled testReadAPIPathLarge, testReadAPIPathLargeWithThresholdParameters, and testReadAPIPathLargeWithThresholdNotMet for TPC using self-contained UNNEST(GENERATE_ARRAY(...)) queries and explicit result set type assertions.
• ITNightlyBigQueryTest: re-enabled testHTAPIWithValidDestinationTableSavesQueriesWithStandardSQL, testReadAPIPathLarge, and testReadAPIPathLargeWithThresholdParameters for TPC.