Skip to content

feat: Add materialization, feature freshness, request latency, and push metrics to feature server#6071

Merged
ntkathole merged 1 commit intofeast-dev:masterfrom
ntkathole:improve_metrics
Mar 9, 2026
Merged

feat: Add materialization, feature freshness, request latency, and push metrics to feature server#6071
ntkathole merged 1 commit intofeast-dev:masterfrom
ntkathole:improve_metrics

Conversation

@ntkathole
Copy link
Copy Markdown
Member

@ntkathole ntkathole commented Mar 5, 2026

What this PR does / why we need it:

Adds rich Prometheus metrics to the Feast feature server covering areas that previously had no observability:

  • Request latency histograms with feature_count and feature_view_count labels so operators can correlate latency with request complexity
  • Materialization tracking — success/failure counters and duration histograms per feature view
  • Feature freshness gauges — staleness (seconds since last materialization) per feature view, updated via background polling
  • Online feature retrieval counters — request counts and entity-row-per-request histograms
  • Push request counters — tracked by push source and mode (online/offline)
  • CPU and memory gauges — moved from inline code to the shared metrics module
    Previously the feature server only had inline CPU/memory monitoring. This PR introduces 10 new metric families covering the full request lifecycle, materialization pipeline, and data freshness.
Screenshot 2026-03-05 at 2 45 51 PM Screenshot 2026-03-05 at 2 46 40 PM

New metrics

Metric Type Labels
feast_feature_server_request_total Counter endpoint, status
feast_feature_server_request_latency_seconds Histogram endpoint, feature_count, feature_view_count
feast_online_features_request_total Counter
feast_online_features_entity_count Histogram
feast_push_request_total Counter push_source, mode
feast_materialization_total Counter feature_view, status
feast_materialization_duration_seconds Histogram feature_view
feast_feature_freshness_seconds Gauge feature_view, project
feast_feature_server_cpu_usage Gauge
feast_feature_server_memory_usage Gauge

Configuration

Metrics are fully opt-in with zero overhead when disabled. Enable via CLI or YAML:

feature_server:
  metrics:
    enabled: true
    resource: true          # CPU / memory gauges
    request: true           # endpoint latency & request counters
    online_features: true   # online feature retrieval counters
    push: true              # push request counters
    materialization: true   # materialization counters & duration
    freshness: true         # per-feature-view freshness gauges

Per-category toggles let disable specific metric groups (e.g., keep CPU/memory but skip request latency). All categories default to true. The --metrics CLI flag works without any YAML config.


Open with Devin

@ntkathole ntkathole requested a review from a team as a code owner March 5, 2026 08:57
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

metrics: true
```

The operator automatically exposes port 8000 and creates the corresponding
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the operator also create the PodMonitor / ServiceMonitor resource owned by the FeatureStore resource for Prometheus operator to discover the metrics endpoint and configure the scraping?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be a good enhancement, we can have CRD detection guard to avoid crashing on vanilla Kubernetes clusters without the Prometheus Operator. Will raise an issue for this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

…sh metrics to feature server

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
@ntkathole ntkathole merged commit 2c6be18 into feast-dev:master Mar 9, 2026
19 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants