Skip to content

fix(bigtable): fix connectivity_error_count - #16656

Open
sushanb wants to merge 1 commit into
googleapis:mainfrom
sushanb:fix/bigtable-connectivity-error-count
Open

sushanb wants to merge 1 commit into
googleapis:mainfrom
sushanb:fix/bigtable-connectivity-error-count

Conversation

@sushanb

@sushanb sushanb commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes three issues in ConnectivityErrorCount and per-attempt TableSchemaMetric label handling:

  1. False-positive & false-negative connectivity error classification: ConnectivityErrorCount::PostCall previously incremented num_errors_ whenever resource_labels_.cluster.empty() || resource_labels_.zone.empty() || IsConnectivityError(...), where IsConnectivityError only checked status.code() != kDeadlineExceeded && !HasServerTiming(client_context). Consequently, successful (OK) RPCs without ResponseParams or server-timing (e.g., DirectPath RPCs carrying bigtable-peer-info, emulator calls, or stripped headers) recorded a connectivity error (1), and non-OK RPCs that reached Bigtable (evidenced by ResponseParams or bigtable-peer-info without server-timing) were also misclassified as connectivity errors.
  2. Per-attempt recording: ConnectivityErrorCount accumulated num_errors_ across retries and emitted a single counter value in OnDone under the final attempt's status/cluster/zone labels instead of recording 1 or 0 per attempt in PostCall with each attempt's own status, cluster, and zone.
  3. Stale cluster/zone labels on failed retries: Per-attempt metrics (AttemptLatency, AttemptLatency2, ServerLatency, and ConnectivityErrorCount) did not clear resource_labels_.cluster and resource_labels_.zone when ResponseParams was absent on a subsequent attempt, leaking the previous attempt's cluster/zone onto retries that failed before reaching a cluster.

Changes

  • Update IsConnectivityError (and IsConnectivityErrorImpl) to require !status.ok() && status.code() != StatusCode::kDeadlineExceeded and the absence of all three server-reached metadata signals (ResponseParams, server-timing, and valid bigtable-peer-info).
  • Record connectivity_error_count (1 or 0) per attempt in ConnectivityErrorCount::PostCall rather than aggregating in OnDone.
  • Clear resource_labels_.cluster and resource_labels_.zone in AttemptLatency::PostCall, AttemptLatency2::PostCall, ServerLatency::PostCall, and ConnectivityErrorCount::PostCall when ResponseParams is absent (while preserving sticky cluster/zone labels on operation-level OperationLatency).
  • Update and expand unit tests in google/cloud/bigtable/internal/table_schema_metrics_test.cc.

@sushanb
sushanb requested a review from a team as a code owner November 11, 2026 21:46
@sushanb
sushanb requested a review from scotthart November 11, 2026 21:46
@product-auto-label product-auto-label Bot added the api: bigtable Issues related to the Bigtable API. label Nov 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the ConnectivityErrorCount metric class to emit metrics per attempt in PostCall instead of accumulating them until OnDone, and ensures that cluster and zone resource labels are cleared across retries when response parameters are missing. It also introduces a helper function IsConnectivityErrorImpl to centralize connectivity error detection. The reviewer feedback suggests factoring out the duplicated logic for updating and clearing resource labels into a shared helper function UpdateResourceLabels in the anonymous namespace, which is repeated across four different PostCall implementations.

@sushanb
sushanb force-pushed the fix/bigtable-connectivity-error-count branch from 5f1aa01 to a5f3a30 Compare November 11, 2026 21:52
@sushanb

sushanb commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors metric collection in table_schema_metrics.cc by introducing helper functions UpdateResourceLabels and IsConnectivityErrorImpl to reduce duplication and simplify logic across latency and connectivity error tracking. It also removes the stateful error counting in ConnectivityErrorCount in favor of recording errors per attempt, and adds comprehensive unit tests to verify cluster/zone resetting behavior and the "three-signal rule" for connectivity errors. The reviewer feedback focuses on optimizing performance by avoiding unnecessary copies of ResponseParams protobuf messages (by passing them by value/moving them) and reducing nesting by returning early.

Record connectivity_error_count per attempt in PostCall using all three server-reached signals (ResponseParams, server-timing, and bigtable-peer-info) on non-OK, non-DEADLINE_EXCEEDED RPCs, and clear per-attempt cluster/zone resource labels when ResponseParams is absent on retries.
@sushanb
sushanb force-pushed the fix/bigtable-connectivity-error-count branch from a5f3a30 to cf38749 Compare November 11, 2026 22:00
@scotthart scotthart added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 11, 2026
@sushanb

sushanb commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the table schema metrics implementation by extracting common logic into helper functions, such as UpdateResourceLabels and IsConnectivityErrorImpl, and simplifies metric tracking for connectivity errors. It also adds comprehensive unit tests to verify cluster/zone stickiness, resetting behavior, and connectivity error detection rules. The reviewer suggested optimizing UpdateResourceLabels by passing response_params as an rvalue reference (&&) to avoid unnecessary move-construction of the std::optional wrapper.

@codecov

codecov Bot commented Nov 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.50%. Comparing base (5dcdef5) to head (cf38749).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16656      +/-   ##
==========================================
- Coverage   92.53%   92.50%   -0.03%     
==========================================
  Files        2269     2269              
  Lines      222896   223302     +406     
==========================================
+ Hits       206250   206567     +317     
- Misses      16646    16735      +89     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable Issues related to the Bigtable API. do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants