Skip to content

feat(http/prom): introduce NewRecordStatusCode middleware #4298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
cratelyn merged 3 commits into from
Nov 17, 2025
Merged

feat(http/prom): introduce NewRecordStatusCode middleware #4298

cratelyn merged 3 commits into from
Nov 17, 2025

Conversation

@cratelyn
Copy link
Member

@cratelyn cratelyn commented Nov 14, 2025 •
edited
Loading

  • refactor(http/prom): status code labelers are Clone
    these types do not contain any internal state, and could thus be
    clonable. this commit makes MkLabelGrpcStatus and
    MkLabelHttpStatus clonable.

    Signed-off-by: katelyn martin kate@buoyant.io

  • refactor(http/prom): constructable RequestCancelled
    this commit removes the uninhabited () state in RequestCancelled.

    this will allow our forthcoming status middleware to construct this
    type when a request is cancelled.

    Signed-off-by: katelyn martin kate@buoyant.io

  • feat(http/prom): add status counting middleware
    this commit introduces a new submodule to linkerd-http-prom.

    this submodule includes a NewRecordStatusCode<N> middleware that can
    be used with the status code labeling implementations in stream_label
    to record Prometheus metrics counting response status codes for HTTP and
    gRPC traffic.

    a test suite demonstrates that this works properly for HTTP and gRPC
    traffic, including edge cases like errors or cancelled bodies.

    Signed-off-by: katelyn martin kate@buoyant.io

@cratelyn cratelyn self-assigned this Nov 14, 2025
@cratelyn cratelyn changed the title kate/decouple record response.pt 1.status code middleware feat(http/prom): introduce NewRecordStatusCode<N> middleware Nov 14, 2025
@cratelyn cratelyn mentioned this pull request Nov 14, 2025
Closed
cratelyn
cratelyn commented Nov 14, 2025
View reviewed changes
linkerd/http/prom/src/status.rs
Comment on lines +261 to +268
let ugh = RequestCancelled.into(); // XXX(kate)

stream_label.end_response(match eos {
EosRef::None => Ok(None),
EosRef::Trailers(trls) => Ok(Some(trls)),
EosRef::Error(error) => Err(error),
EosRef::Cancelled => Err(&ugh),
});
Copy link
Member Author

@cratelyn cratelyn Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: this is one wrinkle that i would like to sort out before this is merged.

Copy link
Member Author

@cratelyn cratelyn Nov 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #4306, which explores altering StreamLabel::end_response() such that it accepts the end-of-stream object directly.

@cratelyn cratelyn marked this pull request as ready for review November 14, 2025 19:09
@cratelyn cratelyn requested a review from a team as a code owner November 14, 2025 19:09
@cratelyn cratelyn force-pushed the kate/decouple-record-response.pt-1.status-code-middleware branch from 34ce5ea to ea7c47b Compare November 14, 2025 19:12
@cratelyn cratelyn changed the base branch from main to kate/app-outbound.refactor-backend-metrics-nits November 14, 2025 19:12
@cratelyn
Copy link
Member Author

cratelyn commented Nov 14, 2025

this is based on #4299.

@cratelyn cratelyn mentioned this pull request Nov 14, 2025
Merged
cratelyn
cratelyn commented Nov 14, 2025
View reviewed changes
linkerd/http/prom/src/stream_label/status.rs
///
/// This generates [`LabelGrpcStatus`] labelers.
#[derive(Debug)]
#[derive(Clone, Debug)]
Copy link
Member Author

@cratelyn cratelyn Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this ended up not being required in more recent iterations of #4300, but seemed like a reasonable and harmless addition.

i'm happy to (a) keep this, (b) back out of this change altogether, or (c) put it in a different pull request. what do you think?

@cratelyn cratelyn changed the title feat(http/prom): introduce NewRecordStatusCode<N> middleware feat(http/prom): introduce NewRecordStatusCode middleware Nov 14, 2025
Base automatically changed from kate/app-outbound.refactor-backend-metrics-nits to main November 17, 2025 18:18
@cratelyn
refactor(http/prom): status code labelers are Clone
3d96dfe 
these types do not contain any internal state, and could thus be
clonable. this commit makes `MkLabelGrpcStatus` and
`MkLabelHttpStatus` clonable.

Signed-off-by: katelyn martin <kate@buoyant.io>
@cratelyn
refactor(http/prom): constructable RequestCancelled
dfe91e1 
this commit removes the uninhabited `()` state in `RequestCancelled`.

this will allow our forthcoming status middleware to construct this
type when a request is cancelled.

Signed-off-by: katelyn martin <kate@buoyant.io>
@cratelyn
feat(http/prom): add status counting middleware
b1b7fe2 
this commit introduces a new submodule to `linkerd-http-prom`.

this submodule includes a `NewRecordStatusCode<N>` middleware that can
be used with the status code labeling implementations in `stream_label`
to record Prometheus metrics counting response status codes for HTTP and
gRPC traffic.

a test suite demonstrates that this works properly for HTTP and gRPC
traffic, including edge cases like errors or cancelled bodies.

Signed-off-by: katelyn martin <kate@buoyant.io>
@cratelyn cratelyn force-pushed the kate/decouple-record-response.pt-1.status-code-middleware branch from ea7c47b to b1b7fe2 Compare November 17, 2025 18:20
@cratelyn
Copy link
Member Author

cratelyn commented Nov 17, 2025 •
edited
Loading

rebased on main now that #4299 has merged.

@cratelyn cratelyn enabled auto-merge (squash) November 17, 2025 18:22
@cratelyn cratelyn merged commit 996eb2f into main Nov 17, 2025
15 checks passed
@cratelyn cratelyn deleted the kate/decouple-record-response.pt-1.status-code-middleware branch November 17, 2025 18:26
cratelyn added a commit that referenced this pull request Nov 17, 2025
@cratelyn
refactor(app/outbound): remove status counters from `NewRecordRespons…
3eab1e8 
…e` (#4300)

**nb:** this branch is based upon
#4299,
and #4298.

in #4299 we made some prepatory
adjustments to the outbound proxy's route-backend metrics layer, and in
#4298 we introduced a new
`linker-http-prom::status` metrics layer that can be used to count response
status codes, in a manner that is agnostic with respect to the particular
protocol that instrumented traffic is using.

this branch performs a sequences of changes oriented towards two concrete
goals: **1:** integrate the `NewRecordStatusCode` middleware into the
outbound proxy's route and backend metrics layers, and **2:** remove status
code measurement from the `NewRecordResponse` middleware.

it's worth stating explicitly that this maintains the existing behavior of the
outbound proxy, and that (1) and (2) must be performed at the same time to
maintain parity.

<img width="498" height="262" alt="image" src="https://github.com/user-attachments/assets/5030e7cd-33df-4cc3-b5cc-5f4db8247bd5" />

to demonstrate that parity is maintained, let's compare metrics from this
branch against a snapshot of metrics exported in `main`.

**🟰 comparing the metrics**

metrics were scraped from the proxy, using a small traffic generation
deployment that runs `curl` against a container running `http-echo`.

a snapshot of the route and backend status/duration counters on main:

```
# HELP outbound_http_route_request_duration_seconds The time between request initialization and response completion.
# TYPE outbound_http_route_request_duration_seconds histogram
# UNIT outbound_http_route_request_duration_seconds seconds
outbound_http_route_request_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 0.5594713199999999
outbound_http_route_request_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
outbound_http_route_request_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
outbound_http_route_request_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
outbound_http_route_request_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
outbound_http_route_request_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
outbound_http_route_request_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 284
# HELP outbound_http_route_request_statuses Completed request-response streams.
# TYPE outbound_http_route_request_statuses counter
outbound_http_route_request_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname="",http_status="200",error=""} 284
# HELP outbound_http_route_backend_response_duration_seconds The time between request completion and response completion.
# TYPE outbound_http_route_backend_response_duration_seconds histogram
# UNIT outbound_http_route_backend_response_duration_seconds seconds
outbound_http_route_backend_response_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 0.4626854930000003
outbound_http_route_backend_response_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="0.025",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="0.1",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="0.25",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
outbound_http_route_backend_response_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 284
# HELP outbound_http_route_backend_response_statuses Completed responses.
# TYPE outbound_http_route_backend_response_statuses counter
outbound_http_route_backend_response_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name="",http_status="200",error=""} 284
```

this was repeated after applying `config.linkerd.io/proxy-version` to use a
proxy with these patches applied.

a snapshot of the route and backend status/duration counters with these
changes:

```
# HELP outbound_http_route_request_duration_seconds The time between request initialization and response completion.
# TYPE outbound_http_route_request_duration_seconds histogram
# UNIT outbound_http_route_request_duration_seconds seconds
outbound_http_route_request_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 0.2557158190000001
outbound_http_route_request_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
outbound_http_route_request_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
outbound_http_route_request_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
outbound_http_route_request_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
outbound_http_route_request_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
outbound_http_route_request_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 128
# HELP outbound_http_route_request_statuses Completed request-response streams.
# TYPE outbound_http_route_request_statuses counter
outbound_http_route_request_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname="",http_status="200",error=""} 128
# HELP outbound_http_route_backend_response_duration_seconds The time between request completion and response completion.
# TYPE outbound_http_route_backend_response_duration_seconds histogram
# UNIT outbound_http_route_backend_response_duration_seconds seconds
outbound_http_route_backend_response_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 0.209469089
outbound_http_route_backend_response_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="0.025",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="0.1",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="0.25",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
outbound_http_route_backend_response_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 128
# HELP outbound_http_route_backend_response_statuses Completed responses.
# TYPE outbound_http_route_backend_response_statuses counter
outbound_http_route_backend_response_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name="",http_status="200",error=""} 128
```

because there are a large number of metrics and labels, a character-level diff
of the two looked like so. as we'd hope, the only differences observed were
values of particular time series. in other words, the same traffic yielded the
same number of time series with the same labels.

```
diff --git a/recorded.txt b/recorded.txt
index 41d4c53..5bf5c13 100644
--- a/recorded.txt
+++ b/recorded.txt
@@ -1,29 +1,29 @@
# HELP outbound_http_route_request_duration_seconds The time between request initialization and response completion.
# TYPE outbound_http_route_request_duration_seconds histogram
# UNIT outbound_http_route_request_duration_seconds seconds
outbound_http_route_request_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} 0.{+2+}55[-94-]71[-3-]{+58+}19[-9999999-]{+0000001+}
outbound_http_route_request_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
outbound_http_route_request_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
outbound_http_route_request_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
outbound_http_route_request_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
outbound_http_route_request_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
outbound_http_route_request_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname=""} {+1+}28[-4-]
# HELP outbound_http_route_request_statuses Completed request-response streams.
# TYPE outbound_http_route_request_statuses counter
outbound_http_route_request_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",hostname="",http_status="200",error=""} {+1+}28[-4-]
# HELP outbound_http_route_backend_response_duration_seconds The time between request completion and response completion.
# TYPE outbound_http_route_backend_response_duration_seconds histogram
# UNIT outbound_http_route_backend_response_duration_seconds seconds
outbound_http_route_backend_response_duration_seconds_sum{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} 0.[-46-]2[-685-]{+09+}4{+6+}9[-30-]0[-00003-]{+89+}
outbound_http_route_backend_response_duration_seconds_count{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="0.025",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="0.05",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="0.1",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="0.25",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="0.5",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="1.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="10.0",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
outbound_http_route_backend_response_duration_seconds_bucket{le="+Inf",parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name=""} {+1+}28[-4-]
# HELP outbound_http_route_backend_response_statuses Completed responses.
# TYPE outbound_http_route_backend_response_statuses counter
outbound_http_route_backend_response_statuses_total{parent_group="core",parent_kind="Service",parent_namespace="simple-app",parent_name="simple-app-v1",parent_port="80",parent_section_name="",route_group="",route_kind="default",route_namespace="",route_name="http",backend_group="core",backend_kind="Service",backend_namespace="simple-app",backend_name="simple-app-v1",backend_port="80",backend_section_name="",http_status="200",error=""} {+1+}28[-4-]
```

---

* feat(app/outbound): add status counters to route metrics

Signed-off-by: katelyn martin <kate@buoyant.io>

* feat(app/outbound): add status counters to backend metrics

Signed-off-by: katelyn martin <kate@buoyant.io>

* feat(app/outbound): add route status counting middleware

Signed-off-by: katelyn martin <kate@buoyant.io>

* feat(app/outbound): add backend status counting middleware

Signed-off-by: katelyn martin <kate@buoyant.io>

* refactor(app/outbound): remove status counter from duration middleware

Signed-off-by: katelyn martin <kate@buoyant.io>

* nit(http/prom): nitpick a todo comment

Signed-off-by: katelyn martin <kate@buoyant.io>

---------

Signed-off-by: katelyn martin <kate@buoyant.io>
cratelyn added a commit that referenced this pull request Dec 1, 2025
@cratelyn
feat(app/inbound): http and grpc status code metrics
0d51e4e 
in #4298, we introduced a new metrics telemetry
layer that can measure and report status codes, in a protocol-agnostic
fashion. this commit integrates this status code telemtry into the
inbound proxy, so that HTTP and gRPC traffic can be observed.

a new family of metrics is introduced to the `InboundMetrics` structure,
and the inbound http\* router's metrics layer is accordingly updated to
thread this metrics family into an extractor, which is in turn provided
to the `NewRecordStatusCode` layer.

\* as a note for reviews, the inbound proxy does not model the http and
grpc protocols are distinct concepts in the network stack's type system,
unlike the outbound proxy. this means that while target types in the
outbound proxy like `Http<()>` are specific to HTTP, the inbound proxy's
distinction of HTTP/gRPC is determined by obtaining and inspecting the
`PermitVariant`.

Signed-off-by: katelyn martin <kate@buoyant.io>
@cratelyn cratelyn mentioned this pull request Dec 1, 2025
Open
cratelyn added a commit that referenced this pull request Dec 1, 2025
@cratelyn
feat(app/inbound): http and grpc status code metrics
9a3b864 
in #4298, we introduced a new metrics telemetry
layer that can measure and report status codes, in a protocol-agnostic
fashion. this commit integrates this status code telemtry into the
inbound proxy, so that HTTP and gRPC traffic can be observed.

a new family of metrics is introduced to the `InboundMetrics` structure,
and the inbound http\* router's metrics layer is accordingly updated to
thread this metrics family into an extractor, which is in turn provided
to the `NewRecordStatusCode` layer.

\* as a note for reviewers, the inbound proxy does not model the http and
grpc protocols as distinct concepts in the network stack's type system,
unlike the outbound proxy. this means that while target types in the
outbound proxy like `Http<()>` are specific to HTTP, the inbound proxy's
distinction of HTTP/gRPC is determined by obtaining and inspecting the
`PermitVariant`.

 #### 🔗 related

some previous pull requests related to this change:

* #4298
* #4180
* #4203
* #4127
* #4119

Signed-off-by: katelyn martin <kate@buoyant.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@olix0r olix0r olix0r approved these changes

Assignees

@cratelyn cratelyn

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cratelyn @olix0r