Skip to content

feat: Added Lineage APIs to get registry objects relationships #5472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
franciscojavierarceo merged 2 commits into from
Jun 30, 2025
Merged

feat: Added Lineage APIs to get registry objects relationships #5472

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
55de37e
feat: Added Lineage APIs to get registry objects relationships
ntkathole Jun 28, 2025
1e8b655
docs: Added docs for rest api endpoints
ntkathole Jun 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
270 changes: 270 additions & 0 deletions docs/reference/feature-servers/registry-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,276 @@ Feast supports running the Registry Server in three distinct modes:
| REST + gRPC | `feast serve_registry --rest-api` | Enables both interfaces |
| REST only | `feast serve_registry --rest-api --no-grpc` | Used for REST-only clients like the UI |

## REST API Endpoints

The REST API provides HTTP/JSON endpoints for accessing all registry metadata. All endpoints are prefixed with `/api/v1` and return JSON responses.

### Authentication

The REST API supports Bearer token authentication. Include your token in the Authorization header:

```bash
Authorization: Bearer <your-token>
```

### Common Query Parameters

Most endpoints support these common query parameters:

- `project` (required for most endpoints): The project name
- `allow_cache` (optional, default: `true`): Whether to allow cached data
- `tags` (optional): Filter results by tags in key=value format

### Entities

#### List Entities
- **Endpoint**: `GET /api/v1/entities`
- **Description**: Retrieve all entities in a project
- **Parameters**:
- `project` (required): Project name
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/entities?project=my_project"
```

#### Get Entity
- **Endpoint**: `GET /api/v1/entities/{name}`
- **Description**: Retrieve a specific entity by name
- **Parameters**:
- `name` (path): Entity name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/entities/user_id?project=my_project"
```

### Data Sources

#### List Data Sources
- **Endpoint**: `GET /api/v1/data_sources`
- **Description**: Retrieve all data sources in a project
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- `tags` (optional): Filter by tags
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/data_sources?project=my_project"
```

#### Get Data Source
- **Endpoint**: `GET /api/v1/data_sources/{name}`
- **Description**: Retrieve a specific data source by name
- **Parameters**:
- `name` (path): Data source name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/data_sources/user_data?project=my_project"
```

### Feature Views

#### List Feature Views
- **Endpoint**: `GET /api/v1/feature_views`
- **Description**: Retrieve all feature views in a project
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- `tags` (optional): Filter by tags
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/feature_views?project=my_project"
```

#### Get Feature View
- **Endpoint**: `GET /api/v1/feature_views/{name}`
- **Description**: Retrieve a specific feature view by name
- **Parameters**:
- `name` (path): Feature view name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/feature_views/user_features?project=my_project"
```

### Feature Services

#### List Feature Services
- **Endpoint**: `GET /api/v1/feature_services`
- **Description**: Retrieve all feature services in a project
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- `tags` (optional): Filter by tags
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/feature_services?project=my_project"
```

#### Get Feature Service
- **Endpoint**: `GET /api/v1/feature_services/{name}`
- **Description**: Retrieve a specific feature service by name
- **Parameters**:
- `name` (path): Feature service name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/feature_services/recommendation_service?project=my_project"
```

### Lineage and Relationships

#### Get Registry Lineage
- **Endpoint**: `GET /api/v1/lineage/registry`
Copy link
Member

@franciscojavierarceo franciscojavierarceo Jun 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably need pagination for big projects at some point in future

- **Description**: Retrieve complete registry lineage with relationships and indirect relationships
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- `filter_object_type` (optional): Filter by object type (`dataSource`, `entity`, `featureView`, `featureService`)
- `filter_object_name` (optional): Filter by object name
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/lineage/registry?project=my_project"
```

#### Get Object Relationships
- **Endpoint**: `GET /api/v1/lineage/objects/{object_type}/{object_name}`
- **Description**: Retrieve relationships for a specific object
- **Parameters**:
- `object_type` (path): Type of object (`dataSource`, `entity`, `featureView`, `featureService`)
- `object_name` (path): Name of the object
- `project` (required): Project name
- `include_indirect` (optional): Whether to include indirect relationships
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/lineage/objects/featureView/user_features?project=my_project&include_indirect=true"
```

#### Get Complete Registry Data
- **Endpoint**: `GET /api/v1/lineage/complete`
- **Description**: Retrieve complete registry data including all objects and relationships (optimized for UI consumption)
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/lineage/complete?project=my_project"
```

### Permissions

#### List Permissions
- **Endpoint**: `GET /api/v1/permissions`
- **Description**: Retrieve all permissions in a project
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/permissions?project=my_project"
```

#### Get Permission
- **Endpoint**: `GET /api/v1/permissions/{name}`
- **Description**: Retrieve a specific permission by name
- **Parameters**:
- `name` (path): Permission name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/permissions/read_features?project=my_project"
```

### Projects

#### List Projects
- **Endpoint**: `GET /api/v1/projects`
- **Description**: Retrieve all projects
- **Parameters**:
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/projects"
```

#### Get Project
- **Endpoint**: `GET /api/v1/projects/{name}`
- **Description**: Retrieve a specific project by name
- **Parameters**:
- `name` (path): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/projects/my_project"
```

### Saved Datasets

#### List Saved Datasets
- **Endpoint**: `GET /api/v1/saved_datasets`
- **Description**: Retrieve all saved datasets in a project
- **Parameters**:
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- `tags` (optional): Filter by tags
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/saved_datasets?project=my_project"
```

#### Get Saved Dataset
- **Endpoint**: `GET /api/v1/saved_datasets/{name}`
- **Description**: Retrieve a specific saved dataset by name
- **Parameters**:
- `name` (path): Saved dataset name
- `project` (required): Project name
- `allow_cache` (optional): Whether to allow cached data
- **Example**:
```bash
curl -H "Authorization: Bearer <token>" \
"http://localhost:6572/api/v1/saved_datasets/training_data?project=my_project"
```

### Response Formats

All endpoints return JSON responses with the following general structure:

- **Success (200)**: Returns the requested data
- **Bad Request (400)**: Invalid parameters or request format
- **Unauthorized (401)**: Missing or invalid authentication token
- **Not Found (404)**: Requested resource does not exist
- **Internal Server Error (500)**: Server-side error

### Interactive API Documentation

When the REST API server is running, you can access interactive documentation at:

- **Swagger UI**: `http://localhost:6572/` (root path)
- **ReDoc**: `http://localhost:6572/docs`
- **OpenAPI Schema**: `http://localhost:6572/openapi.json`

## How to configure the server

Expand Down
40 changes: 40 additions & 0 deletions protos/feast/registry/RegistryServer.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,10 @@ service RegistryServer{
rpc Refresh (RefreshRequest) returns (google.protobuf.Empty) {}
rpc Proto (google.protobuf.Empty) returns (feast.core.Registry) {}

// Lineage RPCs
rpc GetRegistryLineage (GetRegistryLineageRequest) returns (GetRegistryLineageResponse) {}
rpc GetObjectRelationships (GetObjectRelationshipsRequest) returns (GetObjectRelationshipsResponse) {}

}

message RefreshRequest {
Expand Down Expand Up @@ -424,3 +428,39 @@ message DeleteProjectRequest {
string name = 1;
bool commit = 2;
}

// Lineage

message EntityReference {
string type = 1; // "dataSource", "entity", "featureView", "featureService"
string name = 2;
}

message EntityRelation {
EntityReference source = 1;
EntityReference target = 2;
}

message GetRegistryLineageRequest {
string project = 1;
bool allow_cache = 2;
string filter_object_type = 3;
string filter_object_name = 4;
}

message GetRegistryLineageResponse {
repeated EntityRelation relationships = 1;
repeated EntityRelation indirect_relationships = 2;
}

message GetObjectRelationshipsRequest {
string project = 1;
string object_type = 2;
string object_name = 3;
bool include_indirect = 4;
bool allow_cache = 5;
}

message GetObjectRelationshipsResponse {
repeated EntityRelation relationships = 1;
}
2 changes: 2 additions & 0 deletions sdk/python/feast/api/registry/rest/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from feast.api.registry.rest.entities import get_entity_router
from feast.api.registry.rest.feature_services import get_feature_service_router
from feast.api.registry.rest.feature_views import get_feature_view_router
from feast.api.registry.rest.lineage import get_lineage_router
from feast.api.registry.rest.permissions import get_permission_router
from feast.api.registry.rest.projects import get_project_router
from feast.api.registry.rest.saved_datasets import get_saved_dataset_router
Expand All @@ -14,6 +15,7 @@ def register_all_routes(app: FastAPI, grpc_handler):
app.include_router(get_data_source_router(grpc_handler))
app.include_router(get_feature_service_router(grpc_handler))
app.include_router(get_feature_view_router(grpc_handler))
app.include_router(get_lineage_router(grpc_handler))
app.include_router(get_permission_router(grpc_handler))
app.include_router(get_project_router(grpc_handler))
app.include_router(get_saved_dataset_router(grpc_handler))
Loading