Skip to content

Loading AVRO files from local filesystem. #3427

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tseaver merged 4 commits into from
May 16, 2017
Merged

Loading AVRO files from local filesystem. #3427

tseaver merged 4 commits into from
May 16, 2017

Conversation

@tswast
Copy link
Contributor

@tswast tswast commented May 16, 2017

Adds an integration test for loading AVRO files.

Don't set the schema if none is provided (patch).

Closes #3416.

tswast added 3 commits May 16, 2017 13:01
@tswast
Add test avro data.
c27100c 
Generated with `avro-tools`

    $ avro-tools fromjson \
          --schema-file tests/data/colors.avroschema \
          tests/data/colors.json > tests/data/colors.avro

See:
http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
@tswast
Add test to reproduce googleapis#3416.
dcb05b3
@tswast
Don't set schema in load job if none is provided.
2b1f149
@tswast tswast requested a review from tseaver May 16, 2017 21:00
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 16, 2017
@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

@tswast Can you push your branch to this repository, so that the system test actually gets run by the CI before we merge?

@dhermes
Copy link
Contributor

dhermes commented May 16, 2017

@tswast If you don't have push privileges one of us can do it

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

Also, @tweeter0830 reported that the patch did not fix the issue with uploading AVRO data.

@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

I saw that, but I was not able to reproduce the issue after the patch.

@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. labels May 16, 2017
@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

Hmm, I see that this change made the Avro load test pass, but now the CSV load test fails.
https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1920

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

@tswast That is weird, as the testcase clearly sets the schema on the table: it should be passing the schema along.

@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

We missed the "fields" property of the schema object in the patch. Seems to be running fine now. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1922

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

I've pushed the branch here (as tswast-bq-3146) to get the system tests running under CI before merging.

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

BQ system tests are green on Circle CI.

@tseaver tseaver merged commit 4877315 into googleapis:master May 16, 2017
@tswast tswast deleted the bq-3416 branch May 16, 2017 23:14
@tseaver tseaver mentioned this pull request Jun 26, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tseaver tseaver Awaiting requested review from tseaver

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tswast @tseaver @dhermes @googlebot