Skip to content

feat(idea): api_counter metric#18080

Draft
parametalol wants to merge 1 commit intomichael/interceptor-registryfrom
michael/api_counter_metric
Draft

feat(idea): api_counter metric#18080
parametalol wants to merge 1 commit intomichael/interceptor-registryfrom
michael/api_counter_metric

Conversation

@parametalol
Copy link
Copy Markdown
Contributor

@parametalol parametalol commented Dec 5, 2025

Description

Adds real-time counter metrics for API requests, broken down by client profile.

Each incoming API request is matched against built-in client profiles (roxctl, servicenow, splunk_ta, sensor) based on User-Agent headers and request paths. Unmatched requests fall into the unknown profile. A request can match multiple profiles.

Profile trackers are configured via the existing apiRequests metrics configuration group. Descriptor keys are filtered by profile name prefix, so each profile only picks up its own metrics (e.g., a roxctl_total descriptor is handled by the roxctl profile tracker). Labels can reference both common fields (UserID, Status, Method, Path) and profile-specific HTTP headers resolved dynamically via glob patterns (e.g., Rh-Servicenow-InstanceRhServicenowInstance).

The request interceptor handler is registered/unregistered dynamically based on whether any profile tracker is enabled, so there is zero overhead when the feature is disabled.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

Example output from manual testing

The built-in profiles are hardcoded in https://github.com/stackrox/stackrox/blob/michael/api_counter_metric/central/metrics/custom/api_requests/profiles.go

Configuration:

    "metrics": {
      "apiRequests": {
        "descriptors": {
          "roxctl": {
            "labels": [
              "UserID",
              "Status",
              "UserAgent",
              "RhRoxctlCommand"
            ],
          },
          "servicenow": {
            "labels": [
              "UserID",
              "Status",
              "RhServicenowInstance"
            ],
          },
          "servicenow_2": {
            "labels": [
              "UserID",
              "Status",
              "UserAgent"
            ],
          },
          "unknown": {
            "labels": [
              "UserID",
              "Status",
              "UserAgent"
            ],
          }
        },
        "enabled": true
      }
    }
# HELP rox_central_api_request_roxctl The total number of API requests from roxctl aggregated by RhRoxctlCommand, Status, UserAgent, UserID
# TYPE rox_central_api_request_roxctl counter
rox_central_api_request_roxctl{RhRoxctlCommand="central whoami",Status="0",UserAgent="roxctl/4.11.x-172-gb61ea14666-dirty (linux; amd64) grpc-go/1.79.1",UserID="admin"} 2
# HELP rox_central_api_request_servicenow The total number of API requests from servicenow aggregated by RhServicenowInstance, Status, UserID
# TYPE rox_central_api_request_servicenow counter
rox_central_api_request_servicenow{RhServicenowInstance="test-instance",Status="200",UserID="admin"} 3
# HELP rox_central_api_request_servicenow_2 The total number of API requests from servicenow aggregated by Status, UserAgent, UserID
# TYPE rox_central_api_request_servicenow_2 counter
rox_central_api_request_servicenow_2{Status="200",UserAgent="ServiceNow/test-1",UserID="admin"} 3
# HELP rox_central_api_request_unknown The total number of API requests from unknown aggregated by Status, UserAgent, UserID
# TYPE rox_central_api_request_unknown counter
rox_central_api_request_unknown{Status="200",UserAgent="curl/8.15.0",UserID="admin"} 1
rox_central_api_request_unknown{Status="200",UserAgent="kube-probe/1.35; Rox Central/4.11.x-570-g6c0ad01b24 (linux; amd64) grpc-go/1.80.0",UserID=""} 30
rox_central_api_request_unknown{Status="304",UserAgent="Go-http-client/1.1",UserID="mtls:Scanner@7816343570999649157"} 2

Current dependencies on/for this PR:

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Dec 5, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 5, 2025

Codecov Report

❌ Patch coverage is 65.00000% with 91 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.62%. Comparing base (19eec8e) to head (2827cfe).

Files with missing lines Patch % Lines
central/metrics/custom_registry.go 2.85% 31 Missing and 3 partials ⚠️
central/metrics/custom/tracker/tracker_base.go 80.51% 12 Missing and 3 partials ⚠️
central/metrics/custom/api_requests/labels.go 17.64% 14 Missing ⚠️
central/metrics/custom/api_requests/interceptor.go 73.33% 7 Missing and 1 partial ⚠️
central/telemetry/centralclient/client.go 0.00% 6 Missing and 2 partials ⚠️
central/metrics/custom/api_requests/tracker.go 93.33% 4 Missing and 1 partial ⚠️
central/metrics/custom/tracker/configuration.go 28.57% 5 Missing ⚠️
central/metrics/custom/api_requests/profiles.go 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@                       Coverage Diff                        @@
##           michael/interceptor-registry   #18080      +/-   ##
================================================================
+ Coverage                         49.59%   49.62%   +0.03%     
================================================================
  Files                              2767     2771       +4     
  Lines                            208561   208799     +238     
================================================================
+ Hits                             103430   103611     +181     
- Misses                            97455    97506      +51     
- Partials                           7676     7682       +6     
Flag Coverage Δ
go-unit-tests 49.62% <65.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@parametalol parametalol force-pushed the michael/api_counter_metric branch 2 times, most recently from 55a4144 to 35176e7 Compare December 18, 2025 23:29
@parametalol parametalol changed the base branch from master to michael/scoped-and-global-trackers December 18, 2025 23:29
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 35176e7 to 2405451 Compare December 22, 2025 14:33
@parametalol parametalol force-pushed the michael/scoped-and-global-trackers branch from 58a6dae to e6ac4e1 Compare February 2, 2026 16:47
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 5430a52 to 0d43fc3 Compare February 2, 2026 16:47
@parametalol parametalol force-pushed the michael/scoped-and-global-trackers branch from e6ac4e1 to 8da234c Compare March 30, 2026 16:05
Base automatically changed from michael/scoped-and-global-trackers to master March 31, 2026 13:08
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 0d43fc3 to be742d4 Compare April 3, 2026 16:13
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

🚀 Build Images Ready

Images are ready for commit 2827cfe. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-585-g2827cfee76

@parametalol parametalol force-pushed the michael/api_counter_metric branch from 6afc61d to f3cd3b3 Compare April 4, 2026 08:44
@parametalol parametalol changed the base branch from master to michael/interceptor-registry April 4, 2026 18:10
@parametalol parametalol force-pushed the michael/api_counter_metric branch from f3cd3b3 to 8a5cfee Compare April 4, 2026 18:37
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 8a5cfee to d1895a1 Compare April 5, 2026 09:49
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 4d17eb0 to 50de210 Compare April 6, 2026 19:33
@parametalol parametalol marked this pull request as ready for review April 6, 2026 19:46
@parametalol parametalol requested a review from a team as a code owner April 6, 2026 19:47
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="central/metrics/custom/tracker/tracker_base.go" line_range="421-427" />
<code_context>
+// IncrementCounter increments a counter metric with label values extracted from the finding.
+// This should only be called on counter trackers (created with nil generator).
+// If no configuration exists, the increment is a no-op.
+func (tracker *TrackerBase[F]) IncrementCounter(finding F) {
+	if !tracker.isCounter() {
+		// This is a gauge tracker, not a counter tracker
+		return
+	}
+
+	cfg, getters := tracker.getConfiguration()
+	if !cfg.isEnabled() {
+		return
</code_context>
<issue_to_address>
**issue (bug_risk):** Handle nil configuration in IncrementCounter (and other cfg users) to avoid panics and align with the comment

Because `getConfiguration()` can return `nil` before any configuration is applied, calling `cfg.isEnabled()` risks a panic and contradicts the comment that "If no configuration exists, the increment is a no-op".

Please either ensure `getConfiguration` always returns a non-nil `Configuration`, or add a nil check, e.g.:

```go
cfg, getters := tracker.getConfiguration()
if cfg == nil || !cfg.isEnabled() {
    return
}
```

The same nil-safety issue exists in `IsEnabled()` and `Gather()`, so those should be updated similarly if they can be called before `Reconfigure` runs.
</issue_to_address>

### Comment 2
<location path="central/metrics/custom/api_requests/tracker_test.go" line_range="187-196" />
<code_context>
+func TestAllProfilesEmitMetrics(t *testing.T) {
</code_context>
<issue_to_address>
**suggestion (testing):** Consider adding tests that exercise RegisterHandler's add/remove behavior based on tracker enablement

This test validates the full flow from configuration to metrics, but it doesn’t directly exercise `RegisterHandler`’s enable/disable behavior.

Please add unit tests that:

1) Configure all profile trackers as disabled (or omit config), call `RegisterHandler()`, and assert via the interceptor that `handlerName` is not registered.
2) Configure at least one profile tracker as enabled, call `RegisterHandler()`, and assert that `handlerName` is registered.

If the interceptor doesn’t expose this, consider a small test-only accessor or a fake interceptor to verify the registrations directly.

Suggested implementation:

```golang
func TestRegisterHandler_AllProfilesDisabledDoesNotRegisterHandler(t *testing.T) {
	// All trackers disabled (or no enabled profiles) should result in the
	// handler not being registered on the interceptor.
	metrics.DeleteGlobalRegistry()
	interceptor = eventual.New[*requestinterceptor.RequestInterceptor]()
	ri := requestinterceptor.NewRequestInterceptor()
	SetInterceptor(ri)

	// Set up configuration such that all profiles are disabled.
	//
	// This assumes RegisterHandler uses configuration (or equivalent tracker
	// enablement) internally to decide whether to register the handler.
	// Adjust the configuration object to match your actual implementation.
	var allProfilesDisabled bool = false

	// When there are no enabled profiles, RegisterHandler should be a no-op
	// with respect to interceptor handler registration.
	RegisterHandler(allProfilesDisabled)

	// Assert that the handler was not registered.
	// Adjust `handlerName` and the interceptor API used for lookup to match
	// your implementation.
	gotInterceptor, ok := interceptor.Get()
	require.True(t, ok, "interceptor should be set")

	require.False(t, gotInterceptor.HasHandler(handlerName),
		"handler %q should not be registered when all profiles are disabled", handlerName)
}

func TestRegisterHandler_EnabledProfileRegistersHandler(t *testing.T) {
	// When at least one profile is enabled, RegisterHandler should register
	// the handler on the interceptor.
	metrics.DeleteGlobalRegistry()
	interceptor = eventual.New[*requestinterceptor.RequestInterceptor]()
	ri := requestinterceptor.NewRequestInterceptor()
	SetInterceptor(ri)

	// Simulate configuration where at least one profile tracker is enabled.
	var atLeastOneProfileEnabled bool = true

	// When there is at least one enabled profile, RegisterHandler should
	// register the handler with the interceptor.
	RegisterHandler(atLeastOneProfileEnabled)

	gotInterceptor, ok := interceptor.Get()
	require.True(t, ok, "interceptor should be set")

	require.True(t, gotInterceptor.HasHandler(handlerName),
		"handler %q should be registered when at least one profile is enabled", handlerName)
}

func TestAllProfilesEmitMetrics(t *testing.T) {

```

To integrate these tests with your actual codebase, you will likely need to:

1. **Align the `RegisterHandler` signature:**
   - The tests above assume `RegisterHandler` has the signature `func RegisterHandler(enabled bool)`.
   - Update the tests to pass whatever arguments your real `RegisterHandler` requires (e.g., a config struct, tracker list, or group config) and derive the “all disabled” vs. “some enabled” cases from that real configuration shape.
   - If `RegisterHandler` is not exported or has a different name, update the test calls accordingly.

2. **Use the real enablement mechanism:**
   - Replace the `allProfilesDisabled` / `atLeastOneProfileEnabled` booleans with the actual configuration you use to drive tracker enablement.
   - For example, if you drive enablement via `*storage.PrometheusMetrics_Group` or a similar structure, construct:
     - One config where all relevant trackers resolve as disabled or the group is entirely omitted.
     - One config where at least one tracker is enabled.
   - Ensure that `RegisterHandler` uses that config so the tests reflect real behavior.

3. **Align handler registration assertions with your interceptor implementation:**
   - The tests assume:
     - There is a package-level `handlerName` constant that identifies the registered handler.
     - `requestinterceptor.RequestInterceptor` exposes a method `HasHandler(name string) bool` to test registration.
   - If your interceptor instead:
     - Returns a list/map of handler names, replace `HasHandler` with the appropriate lookup (e.g., `Handlers() map[string]http.Handler`).
     - Uses a different identifier or constant for the handler name, adjust `handlerName` accordingly or hard-code the expected name used in `RegisterHandler`.

4. **Imports:**
   - If not already present at the top of `tracker_test.go`, ensure the following imports exist:
     - `github.com/stretchr/testify/require`
     - The `requestinterceptor` package path used elsewhere in this file.
     - The `metrics` and `eventual` packages that are already referenced by `TestAllProfilesEmitMetrics` (reuse the existing imports).

After these adjustments, the two new tests will explicitly validate that `RegisterHandler`’s behavior changes based on tracker enablement and that this is observable via the `RequestInterceptor` handler registrations.
</issue_to_address>

### Comment 3
<location path="pkg/grpc/common/requestinterceptor/request_details_test.go" line_range="96-99" />
<code_context>
 	body := "{ \"n\": 42 }"
 	req, _ := http.NewRequest(http.MethodPost, "/http/body", bytes.NewReader([]byte(body)))
-	rp := getHTTPRequestDetails(context.Background(), req, 0)
+	rp := GetHTTPRequestDetails(context.Background(), req, 0)

 	rb := GetGRPCRequestBody(testBodyI.getTestBody, rp)
</code_context>
<issue_to_address>
**suggestion (testing):** Consider adding tests for GetHTTPRequestDetails around missing identity and edge status codes

Current tests only cover body handling and a successful authenticated request. Please add coverage for:

- `IdentityFromContext` failure, verifying `Path`, `Code`, and headers are still populated while `UserID` stays nil.
- Non-2xx (4xx/5xx) status codes to confirm `Code` is propagated correctly.

A couple of focused tests building requests without identity and with various status codes will help protect the API metrics behavior for `Status` and `UserID` labels.

Suggested implementation:

```golang
func (s *requestDetailsTestSuite) TestHttpWithBody() {
	body := "{ \"n\": 42 }"
	req, _ := http.NewRequest(http.MethodPost, "/http/body", bytes.NewReader([]byte(body)))
	rp := GetHTTPRequestDetails(context.Background(), req, 0)

	rb := GetGRPCRequestBody(testBodyI.getTestBody, rp)
	s.Nil(rb, "body is not captured for HTTP requests")
	req.Header.Add(userAgentHeaderKey, testRP.Headers.Get(userAgentHeaderKey))

	ctx := authn.ContextWithIdentity(context.Background(), testRP.UserID, nil)
	rp = GetHTTPRequestDetails(ctx, req, 200)
	s.Equal(testRP.Path, rp.Path)
	s.Equal(testRP.Code, rp.Code)
	s.Equal(mockID, rp.UserID)
}

func (s *requestDetailsTestSuite) TestGetHTTPRequestDetailsWithoutIdentity() {
	req, _ := http.NewRequest(http.MethodGet, "/http/no-identity", nil)
	req.Header.Add(userAgentHeaderKey, "test-user-agent")

	// No identity in context
	ctx := context.Background()
	rp := GetHTTPRequestDetails(ctx, req, http.StatusOK)

	s.Equal("/http/no-identity", rp.Path, "path should be populated even without identity")
	s.Equal(http.StatusOK, rp.Code, "status code should be propagated")
	s.Nil(rp.UserID, "UserID should be nil when identity is missing")

	// Ensure headers are still captured
	s.Equal("test-user-agent", rp.Headers.Get(userAgentHeaderKey), "headers should be populated even without identity")
}

func (s *requestDetailsTestSuite) TestGetHTTPRequestDetailsNon2xxStatusCodes() {
	statusCodes := []int{
		http.StatusBadRequest,
		http.StatusNotFound,
		http.StatusInternalServerError,
		http.StatusServiceUnavailable,
	}

	for _, code := range statusCodes {
		s.Run(fmt.Sprintf("status_%d", code), func() {
			req, _ := http.NewRequest(http.MethodGet, "/http/status-check", nil)
			ctx := context.Background()

			rp := GetHTTPRequestDetails(ctx, req, code)

			s.Equal("/http/status-check", rp.Path, "path should be correctly captured")
			s.Equal(code, rp.Code, "non-2xx status code should be propagated")
			// UserID should remain nil when no identity is present
			s.Nil(rp.UserID, "UserID should be nil when no identity is provided")
		})
	}

```

These changes assume:
1. `requestDetailsTestSuite` already embeds `suite.Suite` from `testify`, which provides `s.Run`.
2. `fmt` is imported in this test file; if not, add `fmt` to the import list.
3. `GetHTTPRequestDetails` returns a type with `Path`, `Code`, `UserID`, and `Headers` fields, consistent with existing tests.

If `fmt` is not yet imported, add:
- `import "fmt"`
to the existing import block in this file.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@parametalol parametalol force-pushed the michael/interceptor-registry branch from d5efe66 to 19eec8e Compare April 7, 2026 15:45
@parametalol parametalol force-pushed the michael/api_counter_metric branch from 6c0ad01 to 2827cfe Compare April 7, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant