Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 187 additions & 1 deletion docs/reference/feature-servers/registry-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -946,4 +946,190 @@ Please refer the [page](./../registry/registry-permissions.md) for more details

## How to configure Authentication and Authorization ?

Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.

### Metrics

#### Get Resource Counts
- **Endpoint**: `GET /api/v1/metrics/resource_counts`
- **Description**: Retrieve counts of registry objects (entities, data sources, feature views, etc.) for a project or across all projects.
- **Parameters**:
- `project` (optional): Project name to filter resource counts (if not provided, returns counts for all projects)
- **Examples**:
```bash
# Get counts for specific project
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/resource_counts?project=my_project"

# Get counts for all projects
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/resource_counts"
```
- **Response Example** (single project):
```json
{
"project": "my_project",
"counts": {
"entities": 5,
"dataSources": 3,
"savedDatasets": 2,
"features": 12,
"featureViews": 4,
"featureServices": 2
}
}
```
- **Response Example** (all projects):
```json
{
"total": {
"entities": 15,
"dataSources": 8,
"savedDatasets": 5,
"features": 35,
"featureViews": 12,
"featureServices": 6
},
"perProject": {
"project_a": {
"entities": 5,
"dataSources": 3,
"savedDatasets": 2,
"features": 12,
"featureViews": 4,
"featureServices": 2
},
"project_b": {
"entities": 10,
"dataSources": 5,
"savedDatasets": 3,
"features": 23,
"featureViews": 8,
"featureServices": 4
}
}
}
```

#### Get Recently Visited Objects
- **Endpoint**: `GET /api/v1/metrics/recently_visited`
- **Description**: Retrieve the most recently visited registry objects for the authenticated user in a project.
- **Parameters**:
- `project` (optional): Project name to filter recent visits (defaults to current project)
- `object` (optional): Object type to filter recent visits (e.g., entities, features, feature_services)
- `page` (optional): Page number for pagination (starts from 1)
- `limit` (optional): Number of items per page (maximum 100)
- `sort_by` (optional): Field to sort by (e.g., timestamp, path, object)
- `sort_order` (optional): Sort order: "asc" or "desc" (default: "asc")
- **Examples**:
```bash
# Get all recent visits for a project
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project"

# Get recent visits with pagination
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&page=1&limit=10"

# Get recent visits filtered by object type
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&object=entities"

# Get recent visits sorted by timestamp descending
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&sort_by=timestamp&sort_order=desc"
```
- **Response Example** (without pagination):
```json
{
"visits": [
{
"path": "/api/v1/entities/driver",
"timestamp": "2024-07-18T12:34:56.789Z",
"project": "my_project",
"user": "alice",
"object": "entities",
"object_name": "driver",
"method": "GET"
},
{
"path": "/api/v1/feature_services/user_service",
"timestamp": "2024-07-18T12:30:45.123Z",
"project": "my_project",
"user": "alice",
"object": "feature_services",
"object_name": "user_service",
"method": "GET"
}
],
"pagination": {
"totalCount": 2
}
}
```
- **Response Example** (with pagination):
```json
{
"visits": [
{
"path": "/api/v1/entities/driver",
"timestamp": "2024-07-18T12:34:56.789Z",
"project": "my_project",
"user": "alice",
"object": "entities",
"object_name": "driver",
"method": "GET"
}
],
"pagination": {
"page": 1,
"limit": 10,
"totalCount": 25,
"totalPages": 3,
"hasNext": true,
"hasPrevious": false
}
}
```

**Note**: Recent visits are automatically logged when users access registry objects via the REST API. The logging behavior can be configured through the `feature_server.recent_visit_logging` section in `feature_store.yaml` (see configuration section below).

---

## Registry Server Configuration: Recent Visit Logging

The registry server supports configuration of recent visit logging via the `feature_server` section in `feature_store.yaml`.

**Example:**
```yaml
feature_server:
type: local
recent_visit_logging:
limit: 100 # Number of recent visits to store per user
log_patterns:
- ".*/entities/(?!all$)[^/]+$"
- ".*/data_sources/(?!all$)[^/]+$"
- ".*/feature_views/(?!all$)[^/]+$"
- ".*/features/(?!all$)[^/]+$"
- ".*/feature_services/(?!all$)[^/]+$"
- ".*/saved_datasets/(?!all$)[^/]+$"
- ".*/custom_api/.*"
```

**Configuration Options:**
- **recent_visit_logging.limit**: Maximum number of recent visits to store per user (default: 100).
- **recent_visit_logging.log_patterns**: List of regex patterns for API paths to log as recent visits.

**Default Log Patterns:**
- `.*/entities/(?!all$)[^/]+$` - Individual entity endpoints
- `.*/data_sources/(?!all$)[^/]+$` - Individual data source endpoints
- `.*/feature_views/(?!all$)[^/]+$` - Individual feature view endpoints
- `.*/features/(?!all$)[^/]+$` - Individual feature endpoints
- `.*/feature_services/(?!all$)[^/]+$` - Individual feature service endpoints
- `.*/saved_datasets/(?!all$)[^/]+$` - Individual saved dataset endpoints

**Behavior:**
- Only requests matching one of the `log_patterns` will be tracked
- Only the most recent `limit` visits per user are stored
- Metrics endpoints (`/metrics/*`) are automatically excluded from logging to prevent circular references
- Visit data is stored per user and per project in the registry metadata
4 changes: 3 additions & 1 deletion sdk/python/feast/api/registry/rest/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@
from feast.api.registry.rest.feature_views import get_feature_view_router
from feast.api.registry.rest.features import get_feature_router
from feast.api.registry.rest.lineage import get_lineage_router
from feast.api.registry.rest.metrics import get_metrics_router
from feast.api.registry.rest.permissions import get_permission_router
from feast.api.registry.rest.projects import get_project_router
from feast.api.registry.rest.saved_datasets import get_saved_dataset_router


def register_all_routes(app: FastAPI, grpc_handler):
def register_all_routes(app: FastAPI, grpc_handler, server=None):
app.include_router(get_entity_router(grpc_handler))
app.include_router(get_data_source_router(grpc_handler))
app.include_router(get_feature_service_router(grpc_handler))
Expand All @@ -21,3 +22,4 @@ def register_all_routes(app: FastAPI, grpc_handler):
app.include_router(get_permission_router(grpc_handler))
app.include_router(get_project_router(grpc_handler))
app.include_router(get_saved_dataset_router(grpc_handler))
app.include_router(get_metrics_router(grpc_handler, server))
164 changes: 164 additions & 0 deletions sdk/python/feast/api/registry/rest/metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
import json
import logging
from typing import Optional

from fastapi import APIRouter, Depends, Query, Request

from feast.api.registry.rest.rest_utils import (
get_pagination_params,
get_sorting_params,
grpc_call,
paginate_and_sort,
)
from feast.protos.feast.registry import RegistryServer_pb2


def get_metrics_router(grpc_handler, server=None) -> APIRouter:
logger = logging.getLogger(__name__)
router = APIRouter()

@router.get("/metrics/resource_counts", tags=["Metrics"])
async def resource_counts(
project: Optional[str] = Query(
None, description="Project name to filter resource counts"
),
):
def count_resources_for_project(project_name: str):
entities = grpc_call(
grpc_handler.ListEntities,
RegistryServer_pb2.ListEntitiesRequest(project=project_name),
)
data_sources = grpc_call(
grpc_handler.ListDataSources,
RegistryServer_pb2.ListDataSourcesRequest(project=project_name),
)
try:
saved_datasets = grpc_call(
grpc_handler.ListSavedDatasets,
RegistryServer_pb2.ListSavedDatasetsRequest(project=project_name),
)
except Exception:
saved_datasets = {"savedDatasets": []}
try:
features = grpc_call(
grpc_handler.ListFeatures,
RegistryServer_pb2.ListFeaturesRequest(project=project_name),
)
except Exception:
features = {"features": []}
try:
feature_views = grpc_call(
grpc_handler.ListFeatureViews,
RegistryServer_pb2.ListFeatureViewsRequest(project=project_name),
)
except Exception:
feature_views = {"featureViews": []}
try:
feature_services = grpc_call(
grpc_handler.ListFeatureServices,
RegistryServer_pb2.ListFeatureServicesRequest(project=project_name),
)
except Exception:
feature_services = {"featureServices": []}
return {
"entities": len(entities.get("entities", [])),
"dataSources": len(data_sources.get("dataSources", [])),
"savedDatasets": len(saved_datasets.get("savedDatasets", [])),
"features": len(features.get("features", [])),
"featureViews": len(feature_views.get("featureViews", [])),
"featureServices": len(feature_services.get("featureServices", [])),
}

if project:
counts = count_resources_for_project(project)
return {"project": project, "counts": counts}
else:
# List all projects via gRPC
projects_resp = grpc_call(
grpc_handler.ListProjects, RegistryServer_pb2.ListProjectsRequest()
)
all_projects = [
p["spec"]["name"] for p in projects_resp.get("projects", [])
]
all_counts = {}
total_counts = {
"entities": 0,
"dataSources": 0,
"savedDatasets": 0,
"features": 0,
"featureViews": 0,
"featureServices": 0,
}
for project_name in all_projects:
counts = count_resources_for_project(project_name)
all_counts[project_name] = counts
for k in total_counts:
total_counts[k] += counts[k]
return {"total": total_counts, "perProject": all_counts}

@router.get("/metrics/recently_visited", tags=["Metrics"])
async def recently_visited(
request: Request,
project: Optional[str] = Query(
None, description="Project name to filter recent visits"
),
object_type: Optional[str] = Query(
None,
alias="object",
description="Object type to filter recent visits (e.g., entities, features)",
),
pagination_params: dict = Depends(get_pagination_params),
sorting_params: dict = Depends(get_sorting_params),
):
user = None
if hasattr(request.state, "user"):
user = getattr(request.state, "user", None)
if not user:
user = "anonymous"
project_val = project or (server.store.project if server else None)
key = f"recently_visited_{user}"
logger.info(
f"[/metrics/recently_visited] Project: {project_val}, Key: {key}, Object: {object_type}"
)
try:
visits_json = (
server.registry.get_project_metadata(project_val, key)
if server
else None
)
visits = json.loads(visits_json) if visits_json else []
except Exception:
visits = []
if object_type:
visits = [v for v in visits if v.get("object") == object_type]

server_limit = getattr(server, "recent_visits_limit", 100) if server else 100
visits = visits[-server_limit:]

page = pagination_params.get("page", 0)
limit = pagination_params.get("limit", 0)
sort_by = sorting_params.get("sort_by")
sort_order = sorting_params.get("sort_order", "asc")

if page == 0 and limit == 0:
if sort_by:
visits = sorted(
visits,
key=lambda x: x.get(sort_by, ""),
reverse=(sort_order == "desc"),
)
return {"visits": visits, "pagination": {"totalCount": len(visits)}}
else:
if page == 0:
page = 1
if limit == 0:
limit = 50
paged_visits, pagination = paginate_and_sort(
visits, page, limit, sort_by, sort_order
)
return {
"visits": paged_visits,
"pagination": pagination,
}

return router
Loading
Loading