BigQuery: adds models API implementation. #5021

shollyman · 2019-04-25T19:41:37Z

BigQuery: adds models API implementation.

This change adds support for BigQuery ML models as an API resource independent from the previous
ML-As-A-Table variant that existed during beta. It mirrors much of the interactions with Tables, so there's a corresponding ModelId for encapsulating the model reference, ModelInfo for the model metadata, and Model which has service awareness and can be invoked for operations like get/reload.

This initial change does NOT include the extended statistical information from models related to training runs, nor does it include ML feature and labelling column information. That will come in a subsequent change. This change exposes the model type (e.g. LINEAR_REGRESSION), and the ability to do operations like mutate the description/friendlyname/labels.

I've also marked one of the BigQuery methods deprecated due to ambiguous type signature.
boolean delete(String datasetId, String tableId);

When tables were the only kind of resource in a dataset, being able to specify the table via a string was sufficient. However, now that we have independent resource collection kinds within a dataset container, the signature for deletion needs to use more strongly qualified identifiers.

REVIEWER NOTE
I've overridden pom.xml temporarily to pin to an older version of google-api-services-bigquery which will allow testing. The BigQuery service is rolling out the changes necessary so that this no longer needs pinning, but this change is not appropriate for submission until that is complete.

tswast · 2019-04-25T20:06:52Z

...le-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

+   * ModelInfo modelInfo = beforeModel.toBuilder()
+   *     .setDescription(newDescription)
+   *     .build();
+   * Model afterModel = bigquery.update(moddelInfo);


Typo: moddel

Note: We have a little program at https://github.com/googleapis/google-cloud-java/blob/master/utilities/snippets.go for copying code samples into JavaDocs, if you want to try that out.

Fixed the typo. Inclined to add the snippets in a followup and redo the instrumentation then, unless you want this change to get even bigger.

google-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Model.java

tswast · 2019-04-25T20:14:33Z

...clients/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java

+        break;
+      }
+    }
+    assertTrue(found);


You sure you want to assert this since listing resources is eventually consistent? Could be flakey.

Good news, everyone! This should be strongly consistent, per conversation with one of the backend engineers who works on the metadata layer.

tswast · 2019-05-01T21:50:58Z

I've also marked one of the BigQuery methods deprecated due to ambiguous type signature.
boolean delete(String datasetId, String tableId);

Thought: Should this be a separate PR so that it shows up nicely in the changelog?

...le-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

tswast · 2019-05-01T22:24:21Z

...le-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

+    LABELS("labels"),
+    LAST_MODIFIED_TIME("lastModifiedTime"),
+    LOCATION("location"),
+    MODEL_REFERENCE("modelReference"),


I see model type in the Model class, but not here. At first I thought you were waiting for the enum from Apiary, but I guess not?

I called it TYPE, but perhaps MODEL_TYPE is better for the enum?

Oh, you're right. TYPE seems fine. Less redundant.

google-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Model.java

...e-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ModelInfo.java

...lients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java

…1.28.0

tswast · 2019-05-08T21:09:24Z

...le-cloud-clients/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

+    LABELS("labels"),
+    LAST_MODIFIED_TIME("lastModifiedTime"),
+    LOCATION("location"),
+    MODEL_REFERENCE("modelReference"),


Oh, you're right. TYPE seems fine. Less redundant.

codecov · 2019-05-09T16:08:37Z

Codecov Report

Merging #5021 into master will increase coverage by 0.01%.
The diff coverage is 65.09%.

@@             Coverage Diff              @@
##             master    #5021      +/-   ##
============================================
+ Coverage     50.38%    50.4%   +0.01%     
- Complexity    23736    23783      +47     
============================================
  Files          2248     2251       +3     
  Lines        226471   226789     +318     
  Branches      24954    24968      +14     
============================================
+ Hits         114109   114314     +205     
- Misses       103770   103870     +100     
- Partials       8592     8605      +13

Impacted Files	Coverage Δ	Complexity Δ
.../com/google/cloud/bigquery/spi/v2/BigQueryRpc.java	`77.77% <ø> (ø)`	`0 <0> (ø)`	⬇️
.../google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java	`6.1% <0%> (-0.83%)`	`2 <0> (ø)`
...e/cloud/bigquery/testing/RemoteBigQueryHelper.java	`62.5% <0%> (-1.14%)`	`6 <0> (ø)`
...n/java/com/google/cloud/bigquery/BigQueryImpl.java	`80.21% <60.75%> (-3.89%)`	`58 <5> (+5)`
...src/main/java/com/google/cloud/bigquery/Model.java	`75.51% <75.51%> (ø)`	`12 <12> (?)`
...main/java/com/google/cloud/bigquery/ModelInfo.java	`78.84% <78.84%> (ø)`	`18 <18> (?)`
...c/main/java/com/google/cloud/bigquery/ModelId.java	`79.16% <79.16%> (ø)`	`13 <13> (?)`
.../main/java/com/google/cloud/bigquery/BigQuery.java	`78.18% <80.76%> (+0.48%)`	`0 <0> (ø)`	⬇️
...able/gaxx/reframing/ReframingResponseObserver.java	`88.99% <0%> (-1.84%)`	`29% <0%> (-1%)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20912a4...47957b6. Read the comment docs.

shollyman · 2019-05-09T17:35:18Z

The failing kokoro OSX Java 8 test is related to datastore, not BQ. Will proceed with submit.

shollyman and others added 10 commits April 22, 2019 08:14

initial model

ecd1703

Merge branch 'models' of https://github.com/shollyman/google-cloud-java

2b55784

more impl

b890c81

more modelling

2a01f8d

Merge branch 'master' of https://github.com/googleapis/google-cloud-java

6d9aa15

Merge branch 'master' into models

7cb0247

more modelling

b4cd4e6

model testing

cb8961f

more tests and comments

506d44c

more tests, mark bigquery.delete(string, string, string) deprecated

f100375

shollyman added api: bigquery Issues related to the BigQuery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Apr 25, 2019

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Apr 25, 2019

shollyman changed the title ~~Models~~ BigQuery: adds models API implementation. Apr 25, 2019

tswast reviewed Apr 25, 2019

View reviewed changes

shollyman added 3 commits April 29, 2019 10:50

Merge branch 'master' of https://github.com/googleapis/google-cloud-java

3caef2c

Merge branch 'master' into models

b5a6569

typo

715a99a

tswast approved these changes May 1, 2019

View reviewed changes

Merge branch 'master' of https://github.com/googleapis/google-cloud-java

ca1f0e0

yoshi-automation added the 🚨 This issue needs some love. label May 2, 2019

shollyman added 3 commits May 8, 2019 10:52

Merge branch 'master' of https://github.com/googleapis/google-cloud-java

7896589

Merge branch 'master' into models and unpin

bf6fa35

address review comments

16d3ab8

shollyman marked this pull request as ready for review May 8, 2019 18:06

shollyman requested a review from a team as a code owner May 8, 2019 18:06

shollyman added 3 commits May 8, 2019 11:22

formatter + typo

83769ce

update pinning of bigquery in google-cloud-clients to v2-rev20190423-…

47c1fca

…1.28.0

step down from 1.28.0 to 1.27.0 due to bounding on deps

ae12428

tswast approved these changes May 8, 2019

View reviewed changes

shollyman added 3 commits May 9, 2019 09:07

backout local pin, will pick up new pin from master

dc0cc2c

Merge branch 'master' of https://github.com/googleapis/google-cloud-java

09913dd

Merge branch 'master' into models

47957b6

shollyman merged commit 57ac06a into googleapis:master May 9, 2019

shollyman removed 🚨 This issue needs some love. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels May 9, 2019

chingor13 mentioned this pull request May 15, 2019

Release google-cloud-java v0.92.0 #5189

Merged

BigQuery: adds models API implementation. #5021

BigQuery: adds models API implementation. #5021

Uh oh!

Conversation

shollyman commented Apr 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tswast commented May 1, 2019

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 9, 2019

Codecov Report

Uh oh!

shollyman commented May 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants