Bigtable: instrument client with Opencensus metrics #6143

igorbernstein2 · 2019-08-22T22:38:25Z

It adds stats to logical operations for things like:

how long did logically method call take across all of its retry attempts
whats the distribution of rows read per operation
whats the distribution of the number of entries in a bulk mutation

This should allow users to better debug the performance of the client. Allowing to debug scenarios like high perceived latency on the client side, but no latency visible on the serverside due to retries.

This also adds a more user friendly way to enable data collection by exposing a method on the settings class.

Main changes:

Follow grpc's example and create Rpc*Constant files to configure opencensus stat views
Mark the constants as internal and expose them via a static method on the settings
Create callables for unary chains and special callables for ReadRows & MutateRows
Push metric & tracing instrumentation to the leaf callables

# Conflicts: # google-cloud-clients/google-cloud-bigtable/pom.xml

...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java

* clean up code

codecov · 2019-08-27T20:30:24Z

Codecov Report

Merging #6143 into master will increase coverage by 0.3%.
The diff coverage is 67.81%.

@@             Coverage Diff             @@
##             master    #6143     +/-   ##
===========================================
+ Coverage     47.23%   47.53%   +0.3%     
- Complexity    27458    27470     +12     
===========================================
  Files          2526     2534      +8     
  Lines        274535   274918    +383     
  Branches      30579    31408    +829     
===========================================
+ Hits         129668   130674   +1006     
- Misses       134446   134611    +165     
+ Partials      10421     9633    -788

Impacted Files	Coverage Δ	Complexity Δ
...le/cloud/bigtable/data/v2/models/BulkMutation.java	`97.29% <0%> (-2.71%)`	`9 <0> (ø)`
...igtable/data/v2/stub/metrics/RpcViewConstants.java	`0% <0%> (ø)`	`0 <0> (?)`
...e/cloud/bigtable/data/v2/BigtableDataSettings.java	`32.14% <0%> (-1.2%)`	`4 <0> (ø)`
.../cloud/bigtable/data/v2/stub/metrics/RpcViews.java	`0% <0%> (ø)`	`0 <0> (?)`
...le/data/v2/stub/metrics/MeasuredUnaryCallable.java	`100% <100%> (ø)`	`2 <2> (?)`
...data/v2/stub/metrics/MeasuredReadRowsCallable.java	`100% <100%> (ø)`	`2 <2> (?)`
...ta/v2/stub/metrics/MeasuredMutateRowsCallable.java	`100% <100%> (ø)`	`2 <2> (?)`
...ud/bigtable/data/v2/stub/EnhancedBigtableStub.java	`95.45% <100%> (+0.6%)`	`21 <3> (-1)`	⬇️
.../v2/stub/metrics/MeasuredMutateRowsCallableV2.java	`25% <25%> (ø)`	`1 <1> (?)`
...ogle/cloud/bigtable/data/v2/stub/metrics/Util.java	`60% <60%> (ø)`	`5 <5> (?)`
... and 194 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb526b4...48a2acb. Read the comment docs.

kolea2 · 2019-08-29T20:05:21Z

google-cloud-clients/google-cloud-bigtable/README.md

+  of mutations per BulkMutation.
+
+
+By default, the functionality is disabled. To enable, you need to add a couple of


I would reword: "To enable, please follow the instructions below:"

changed the wording, PTAL

kolea2 · 2019-08-29T20:05:37Z

google-cloud-clients/google-cloud-bigtable/README.md

+libraryDependencies += "io.opencensus" % "opencensus-exporter-stats-stackdriver" % "0.23.0"
+```
+
+Then at the start of your application configure the exporter and enable the Bigtable stats views:


Remove "then"

kolea2 · 2019-08-29T20:07:04Z

...gle-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataSettings.java

  }

+  /**
+   * Enables Opencensus metric aggregations.


minor - change all occurrences of "Opencensus" to "OpenCensus"

kolea2 · 2019-08-29T20:07:27Z

...gle-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataSettings.java

+   * }</pre>
+   */
+  @BetaApi("Opencensus stats integration is currently unstable and may change in the future")
+  public static void enableOpencensusStats() {


kolea2 · 2019-08-29T20:09:00Z

...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java

+   *   <li>Add tracing & metrics.
   * </ul>
   */
+  @Deprecated


point to new method?

kolea2 · 2019-08-29T20:18:46Z

...src/main/java/com/google/cloud/bigtable/data/v2/stub/metrics/MeasuredMutateRowsCallable.java

+import javax.annotation.Nonnull;
+
+/**
+ * @deprecated Please use {@link MeasuredMutateRowsCallableV2}.


please add an internal use only comment as well

WIP: Bigtable: instrument client with Opencensus metrics

fdf32a3

igorbernstein2 added do not merge Indicates a pull request not ready for merge, due to either quality or timing. api: bigtable Issues related to the Bigtable API. labels Aug 22, 2019

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 22, 2019

igorbernstein2 added 2 commits August 26, 2019 19:58

Add tests & bug fixes

f511bca

tweaks

775b7e8

igorbernstein2 changed the title ~~WIP: Bigtable: instrument client with Opencensus metrics~~ Bigtable: instrument client with Opencensus metrics Aug 27, 2019

igorbernstein2 marked this pull request as ready for review August 27, 2019 00:12

igorbernstein2 requested review from kolea2 and rahulKQL August 27, 2019 00:12

igorbernstein2 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019

Merge remote-tracking branch 'origin/master' into metrics

af060ef

# Conflicts: # google-cloud-clients/google-cloud-bigtable/pom.xml

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019

igorbernstein2 added 2 commits August 26, 2019 20:35

Update README

c46b7fc

add a note to try to keep the readme up to date

e9261fe

rahulKQL reviewed Aug 27, 2019

View reviewed changes

igorbernstein2 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019

igorbernstein2 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Aug 27, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019

igorbernstein2 mentioned this pull request Aug 27, 2019

Add support for DropWizard Metrics #1209

Closed

* Fix method tag names

1606cf1

* clean up code

igorbernstein2 added 6 commits August 27, 2019 17:20

tweak opencensus identifiers

e3081ce

tweak opencensus identifiers

ac72c50

tweak opencensus identifiers

62a03bb

Merge remote-tracking branch 'origin/master' into metrics

7667c19

fix merge

af38794

integrate with new batching api

9c11049

igorbernstein2 added 3 commits August 29, 2019 14:08

Merge remote-tracking branch 'origin/master' into metrics

39f7fd4

make metric names a bit more consistent

daf6138

fix readme scope

6ee753b

igorbernstein2 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019

more polish

dd96b44

kolea2 reviewed Aug 29, 2019

View reviewed changes

kolea2 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Aug 29, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019

igorbernstein2 added 2 commits August 29, 2019 17:50

fix readme deps

35482d5

fix readme deps

48a2acb

igorbernstein2 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Aug 29, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019

kolea2 approved these changes Aug 30, 2019

View reviewed changes

igorbernstein2 mentioned this pull request Aug 30, 2019

Integrating with stackdriver leads to dependency hell #6203

Open

igorbernstein2 merged commit 8f6aac8 into googleapis:master Sep 3, 2019

igorbernstein2 deleted the metrics branch September 3, 2019 22:19

chingor13 mentioned this pull request Sep 5, 2019

Release google-cloud-java v0.108.0 #6223

Merged

		of mutations per BulkMutation.


		By default, the functionality is disabled. To enable, you need to add a couple of

Bigtable: instrument client with Opencensus metrics #6143

Bigtable: instrument client with Opencensus metrics #6143

Uh oh!

Conversation

igorbernstein2 commented Aug 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

igorbernstein2 commented Aug 22, 2019 •

edited

Loading

codecov bot commented Aug 27, 2019 •

edited

Loading