Skip to content

Job to infer schema from BigQuerySource with query not running on billing project id #3416

@sudohainguyen

Description

@sudohainguyen

Expected Behavior

when using BigQuery Batch source with the given query instead of table name, I suppose the schema would be inferred internally by executing jobs on billing project id (if provided).

Current Behavior

The inference job is running on the attached project id with the current service account even when I provided specific billing project id.

Steps to reproduce

  1. Prepare a repo with provided billing project id (named A) which is different from the attached one (named B).
  2. Create a BigQuerySource with query statements.
  3. feast apply
  4. Now check where the job running -> it's B

Specifications

  • Version: 0.27.1
  • Platform: Linux
  • Subsystem: Ubuntu

Possible Solution

a few changes needed here line 162

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions