Skip to content

Commit b7ae488

Browse files
aniketpaluaniketpalu
authored
feat(server): Remote Materialization (#6649)
* feat(server): Add async materialization via ?async=true query param When ?async=true is passed to /materialize and /materialize-incremental, the endpoint fires off materialization in a background thread and returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Without ?async=true, behavior is unchanged. Client-side additions: - remote=True param on store.materialize() / materialize_incremental() to delegate to the feature server (URL/TLS from online_store config) - wait=False support with store.poll_materialization() for status polling - FeatureView state set to MATERIALIZING before 202, reset on failure Server-side additions: - ?async=true on existing /materialize and /materialize-incremental - ?force=true to override stuck MATERIALIZING state - Four module-level helpers for testability: _authorize_materialize_views, _check_already_materializing, _update_fv_state, _parse_materialize_timestamps Registry fixes: - SQL and Snowflake registries now set FV state to AVAILABLE_ONLINE in apply_materialization() (parity with file-based registry) Addresses #4526 Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * feat(server): Add async materialization via ?async=true query param When ?async=true is passed to /materialize and /materialize-incremental, the endpoint fires off materialization in a background thread and returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Without ?async=true, behavior is unchanged. Client-side additions: - remote=True param on store.materialize() / materialize_incremental() to delegate to the feature server (URL/TLS from online_store config) - wait=False support with store.poll_materialization() for status polling - FeatureView state set to MATERIALIZING before 202, reset on failure Server-side additions: - ?async=true on existing /materialize and /materialize-incremental - ?force=true to override stuck MATERIALIZING state - Four module-level helpers for testability: _authorize_materialize_views, _check_already_materializing, _update_fv_state, _parse_materialize_timestamps Registry fixes: - SQL and Snowflake registries now set FV state to AVAILABLE_ONLINE in apply_materialization() (parity with file-based registry) Addresses #4526 Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * feat(sdk): Auto-delegate materialize to feature server when online_store is remote When online_store.type == remote, materialize() and materialize_incremental() POST to the feature server with ?async=true (fire-and-forget) instead of running the engine locally. Optional force=True maps to ?force=true for stuck MATERIALIZING recovery. URL/TLS/auth come from online_store config. Shared _delegate_remote_materialize() builds query params and posts; RemoteComputeEngine remains a follow-up for provider-layer remoting. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix(server): Let store own MATERIALIZING transitions on async path Remove server-side MATERIALIZING pre-set before 202 which conflicted with store.materialize() state machine (MATERIALIZING → MATERIALIZING rejected). Async now accepts and runs materialize in the background; store owns transitions. ?force=true resets stuck MATERIALIZING FVs to GENERATED so normal materialize can proceed. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: Address #6649 review — run_async, race reserve, version threading - Revert FeatureView.state from __eq__ - Add run_async for remote sync vs async HTTP - Reserve MATERIALIZING before 202; idempotent store transitions - Thread version through authorize and all materialize server paths - Narrow silent excepts; document version on request models Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix(server): Only reset MATERIALIZING FVs on async materialize failure Avoid wiping AVAILABLE_ONLINE when a post-success exception fires (e.g. SparkApp CR already cleaned up). Reuse the stuck-state helper so true failures still return to GENERATED. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: Default run_async to False and cover version forwarding Preserve sync semantics for remote users; opt into fire-and-forget with run_async=True. Add unit tests that version is threaded through authorize and both sync/async materialize server paths. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix(server): Use dedicated executor for async materialize Isolate long materialize waits from the default/shared thread pool used by online serving. Pool size via FEAST_MATERIALIZE_MAX_WORKERS (default 2); shut down on app lifespan exit without blocking. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> --------- Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
1 parent e8edab5 commit b7ae488

5 files changed

Lines changed: 619 additions & 58 deletions

File tree

‎sdk/python/feast/feature_server.py‎

Lines changed: 251 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import time
2121
import traceback
2222
from collections import defaultdict
23+
from concurrent.futures import ThreadPoolExecutor
2324
from contextlib import asynccontextmanager
2425
from datetime import datetime
2526
from importlib import resources as importlib_resources
@@ -31,6 +32,7 @@
3132
from fastapi import (
3233
Depends,
3334
FastAPI,
35+
Query,
3436
Request,
3537
Response,
3638
WebSocket,
@@ -41,7 +43,7 @@
4143
from fastapi.logger import logger
4244
from fastapi.responses import JSONResponse
4345
from fastapi.staticfiles import StaticFiles
44-
from pydantic import BaseModel, field_validator
46+
from pydantic import BaseModel, Field, field_validator
4547

4648
import feast
4749
from feast import metrics as feast_metrics
@@ -54,6 +56,7 @@
5456
)
5557
from feast.feast_object import FeastObject
5658
from feast.feature_server_utils import convert_response_to_dict
59+
from feast.feature_view import FeatureViewState
5760
from feast.feature_view_utils import get_feature_view_from_feature_store
5861
from feast.filter_models import ComparisonFilter, CompoundFilter
5962
from feast.permissions.action import WRITE, AuthzedAction
@@ -94,12 +97,26 @@ class MaterializeRequest(BaseModel):
9497
feature_views: Optional[List[str]] = None
9598
disable_event_timestamp: bool = False
9699
full_feature_names: bool = False
100+
version: Optional[str] = Field(
101+
None,
102+
description=(
103+
"Optional version to materialize (e.g. 'v2'). Requires feature_views "
104+
"with exactly one entry and registry.enable_online_feature_view_versioning."
105+
),
106+
)
97107

98108

99109
class MaterializeIncrementalRequest(BaseModel):
100110
end_ts: str
101111
feature_views: Optional[List[str]] = None
102112
full_feature_names: bool = False
113+
version: Optional[str] = Field(
114+
None,
115+
description=(
116+
"Optional version to materialize (e.g. 'v2'). Requires feature_views "
117+
"with exactly one entry and registry.enable_online_feature_view_versioning."
118+
),
119+
)
103120

104121

105122
class GetOnlineFeaturesRequest(BaseModel):
@@ -333,6 +350,128 @@ async def load_static_artifacts(app: FastAPI, store):
333350
logger.warning(f"Failed to load static artifacts: {e}")
334351

335352

353+
def _authorize_materialize_views(
354+
store: "feast.FeatureStore",
355+
feature_view_names: Optional[List[str]],
356+
version: Optional[str] = None,
357+
) -> List[str]:
358+
"""Resolve + authorize feature views for materialization.
359+
360+
Returns the resolved list of FV names (all eligible FVs when
361+
feature_view_names is None).
362+
"""
363+
parsed_version = store._validate_materialize_version(version, feature_view_names)
364+
feature_views_to_materialize = store._get_feature_views_to_materialize(
365+
feature_view_names, version=parsed_version
366+
)
367+
for fv in feature_views_to_materialize:
368+
assert_permissions(
369+
resource=fv,
370+
actions=[AuthzedAction.WRITE_ONLINE],
371+
)
372+
return [fv.name for fv in feature_views_to_materialize]
373+
374+
375+
def _check_already_materializing(
376+
store: "feast.FeatureStore",
377+
fv_names: List[str],
378+
) -> Optional[JSONResponse]:
379+
"""Return a 409 JSONResponse if any requested FV is already MATERIALIZING."""
380+
conflicting: List[str] = []
381+
for fv_name in fv_names:
382+
try:
383+
fv = store.registry.get_feature_view(
384+
fv_name, store.project, allow_cache=False
385+
)
386+
if getattr(fv, "state", None) == FeatureViewState.MATERIALIZING:
387+
conflicting.append(fv_name)
388+
except (FeatureViewNotFoundException, KeyError):
389+
pass
390+
except Exception as e:
391+
logger.warning(
392+
f"Unexpected error checking MATERIALIZING state for {fv_name}: {e}"
393+
)
394+
if conflicting:
395+
return JSONResponse(
396+
status_code=409,
397+
content={
398+
"error": (
399+
f"Cannot start async materialization — the following feature "
400+
f"views are already in MATERIALIZING state: {conflicting}. "
401+
f"Use ?force=true to override."
402+
),
403+
"feature_views": conflicting,
404+
},
405+
)
406+
return None
407+
408+
409+
def _update_fv_state(
410+
store: "feast.FeatureStore",
411+
fv_names: List[str],
412+
state: FeatureViewState,
413+
) -> None:
414+
"""Set FV state in the registry for each named feature view."""
415+
for fv_name in fv_names:
416+
try:
417+
fv = store.registry.get_feature_view(
418+
fv_name, store.project, allow_cache=False
419+
)
420+
fv.state = state
421+
store.registry.apply_feature_view(fv, store.project)
422+
except (FeatureViewNotFoundException, KeyError):
423+
logger.warning(f"Feature view {fv_name} not found; skip state={state}")
424+
except Exception as e:
425+
logger.warning(f"Failed to set state={state} for {fv_name}: {e}")
426+
427+
428+
def _reset_stuck_materializing_to_generated(
429+
store: "feast.FeatureStore",
430+
fv_names: List[str],
431+
) -> None:
432+
"""Reset FVs currently in MATERIALIZING to GENERATED (force override)."""
433+
stuck: List[str] = []
434+
for fv_name in fv_names:
435+
try:
436+
fv = store.registry.get_feature_view(
437+
fv_name, store.project, allow_cache=False
438+
)
439+
if getattr(fv, "state", None) == FeatureViewState.MATERIALIZING:
440+
stuck.append(fv_name)
441+
except (FeatureViewNotFoundException, KeyError):
442+
pass
443+
except Exception as e:
444+
logger.warning(f"Unexpected error while force-resetting {fv_name}: {e}")
445+
if stuck:
446+
_update_fv_state(store, stuck, FeatureViewState.GENERATED)
447+
logger.info(
448+
"Force reset MATERIALIZING → GENERATED for feature views: %s", stuck
449+
)
450+
451+
452+
def _parse_materialize_timestamps(
453+
request: "MaterializeRequest",
454+
) -> tuple:
455+
"""Parse and validate start/end timestamps from a MaterializeRequest."""
456+
if request.disable_event_timestamp:
457+
now = datetime.now()
458+
return datetime(1970, 1, 1), now
459+
460+
if not request.start_ts or not request.end_ts:
461+
raise ValueError(
462+
"start_ts and end_ts are required when disable_event_timestamp is False"
463+
)
464+
try:
465+
start_date = utils.make_tzaware(parser.parse(request.start_ts))
466+
end_date = utils.make_tzaware(parser.parse(request.end_ts))
467+
except (ValueError, TypeError) as e:
468+
raise ValueError(f"Invalid timestamp format: {e}") from e
469+
470+
if start_date >= end_date:
471+
raise ValueError(f"start_ts ({start_date}) must be before end_ts ({end_date})")
472+
return start_date, end_date
473+
474+
336475
def get_app(
337476
store: "feast.FeatureStore",
338477
registry_ttl_sec: int = DEFAULT_FEATURE_SERVER_REGISTRY_TTL,
@@ -412,6 +551,22 @@ def get_app(
412551
else:
413552
logger.debug("Offline write batching is DISABLED")
414553

554+
# Dedicated pool for async materialize so long Spark/offline waits do not
555+
# starve the default executor used by online serving and run_in_threadpool.
556+
_mat_workers_raw = os.environ.get("FEAST_MATERIALIZE_MAX_WORKERS", "2")
557+
try:
558+
materialize_max_workers = max(1, int(_mat_workers_raw))
559+
except ValueError:
560+
logger.warning(
561+
"Invalid FEAST_MATERIALIZE_MAX_WORKERS=%r; using default 2",
562+
_mat_workers_raw,
563+
)
564+
materialize_max_workers = 2
565+
materialize_executor = ThreadPoolExecutor(
566+
max_workers=materialize_max_workers,
567+
thread_name_prefix="feast-materialize",
568+
)
569+
415570
def stop_refresh():
416571
nonlocal shutting_down
417572
shutting_down = True
@@ -445,6 +600,9 @@ async def lifespan(app: FastAPI):
445600
stop_refresh()
446601
if offline_batcher is not None:
447602
offline_batcher.shutdown()
603+
# wait=False: do not block process exit on in-flight materialize
604+
# (same fire-and-forget contract as returning 202 mid-job).
605+
materialize_executor.shutdown(wait=False)
448606
await store.close()
449607

450608
app = FastAPI(lifespan=lifespan)
@@ -798,70 +956,115 @@ async def chat_ui():
798956
return Response(content=content, media_type="text/html")
799957

800958
@app.post("/materialize", dependencies=[Depends(inject_user_details)])
801-
async def materialize(request: MaterializeRequest) -> None:
959+
async def materialize(
960+
request: MaterializeRequest,
961+
async_mode: bool = Query(False, alias="async"),
962+
force: bool = Query(False),
963+
):
802964
with feast_metrics.track_request_latency("/materialize"):
803-
if request.feature_views:
804-
for feature_view in request.feature_views:
805-
resource = await _get_feast_object(feature_view, True)
806-
assert_permissions(
807-
resource=resource,
808-
actions=[AuthzedAction.WRITE_ONLINE],
809-
)
810-
else:
811-
feature_views_to_materialize = store._get_feature_views_to_materialize(
812-
None
813-
)
814-
for fv in feature_views_to_materialize:
815-
assert_permissions(
816-
resource=fv,
817-
actions=[AuthzedAction.WRITE_ONLINE],
818-
)
965+
fv_names = _authorize_materialize_views(
966+
store, request.feature_views, version=request.version
967+
)
968+
start_date, end_date = _parse_materialize_timestamps(request)
819969

820-
if request.disable_event_timestamp:
821-
now = datetime.now()
822-
start_date = datetime(1970, 1, 1)
823-
end_date = now
824-
else:
825-
if not request.start_ts or not request.end_ts:
826-
raise ValueError(
827-
"start_ts and end_ts are required when disable_event_timestamp is False"
828-
)
829-
start_date = utils.make_tzaware(parser.parse(request.start_ts))
830-
end_date = utils.make_tzaware(parser.parse(request.end_ts))
970+
if async_mode:
971+
if force:
972+
_reset_stuck_materializing_to_generated(store, fv_names)
973+
else:
974+
conflict = _check_already_materializing(store, fv_names)
975+
if conflict:
976+
return conflict
977+
978+
# Reserve MATERIALIZING before 202 so concurrent requests hit 409.
979+
# store.materialize() treats already-MATERIALIZING as a no-op.
980+
_update_fv_state(store, fv_names, FeatureViewState.MATERIALIZING)
981+
982+
def _run_materialize():
983+
try:
984+
store.materialize(
985+
start_date,
986+
end_date,
987+
fv_names,
988+
disable_event_timestamp=request.disable_event_timestamp,
989+
full_feature_names=request.full_feature_names,
990+
version=request.version,
991+
)
992+
except Exception as e:
993+
logger.error(
994+
f"Async materialization failed for {fv_names}: {e}",
995+
exc_info=True,
996+
)
997+
_reset_stuck_materializing_to_generated(store, fv_names)
998+
999+
loop = asyncio.get_running_loop()
1000+
loop.run_in_executor(materialize_executor, _run_materialize)
1001+
1002+
return JSONResponse(
1003+
status_code=202,
1004+
content={"status": "accepted", "feature_views": fv_names},
1005+
)
8311006

8321007
await run_in_threadpool(
8331008
store.materialize,
8341009
start_date,
8351010
end_date,
836-
request.feature_views,
1011+
fv_names,
8371012
disable_event_timestamp=request.disable_event_timestamp,
8381013
full_feature_names=request.full_feature_names,
1014+
version=request.version,
8391015
)
8401016

8411017
@app.post("/materialize-incremental", dependencies=[Depends(inject_user_details)])
842-
async def materialize_incremental(request: MaterializeIncrementalRequest) -> None:
1018+
async def materialize_incremental(
1019+
request: MaterializeIncrementalRequest,
1020+
async_mode: bool = Query(False, alias="async"),
1021+
force: bool = Query(False),
1022+
):
8431023
with feast_metrics.track_request_latency("/materialize-incremental"):
844-
if request.feature_views:
845-
for feature_view in request.feature_views:
846-
resource = await _get_feast_object(feature_view, True)
847-
assert_permissions(
848-
resource=resource,
849-
actions=[AuthzedAction.WRITE_ONLINE],
850-
)
851-
else:
852-
feature_views_to_materialize = store._get_feature_views_to_materialize(
853-
None
1024+
fv_names = _authorize_materialize_views(
1025+
store, request.feature_views, version=request.version
1026+
)
1027+
end_date = utils.make_tzaware(parser.parse(request.end_ts))
1028+
1029+
if async_mode:
1030+
if force:
1031+
_reset_stuck_materializing_to_generated(store, fv_names)
1032+
else:
1033+
conflict = _check_already_materializing(store, fv_names)
1034+
if conflict:
1035+
return conflict
1036+
1037+
_update_fv_state(store, fv_names, FeatureViewState.MATERIALIZING)
1038+
1039+
def _run_materialize_incremental():
1040+
try:
1041+
store.materialize_incremental(
1042+
end_date,
1043+
fv_names,
1044+
full_feature_names=request.full_feature_names,
1045+
version=request.version,
1046+
)
1047+
except Exception as e:
1048+
logger.error(
1049+
f"Async materialize-incremental failed for {fv_names}: {e}",
1050+
exc_info=True,
1051+
)
1052+
_reset_stuck_materializing_to_generated(store, fv_names)
1053+
1054+
loop = asyncio.get_running_loop()
1055+
loop.run_in_executor(materialize_executor, _run_materialize_incremental)
1056+
1057+
return JSONResponse(
1058+
status_code=202,
1059+
content={"status": "accepted", "feature_views": fv_names},
8541060
)
855-
for fv in feature_views_to_materialize:
856-
assert_permissions(
857-
resource=fv,
858-
actions=[AuthzedAction.WRITE_ONLINE],
859-
)
1061+
8601062
await run_in_threadpool(
8611063
store.materialize_incremental,
862-
utils.make_tzaware(parser.parse(request.end_ts)),
863-
request.feature_views,
1064+
end_date,
1065+
fv_names,
8641066
full_feature_names=request.full_feature_names,
1067+
version=request.version,
8651068
)
8661069

8671070
@app.exception_handler(Exception)

0 commit comments

Comments
 (0)