Expected Behavior
write_to_offline_store should resolve the feature view with one registry lookup,
regardless of which feature view type the name refers to.
Current Behavior
FeatureStore.write_to_offline_store resolves the feature view by trying each
getter in turn and catching FeatureViewNotFoundException
(sdk/python/feast/feature_store.py, lines 3650-3663 on master; the method
begins at 3637):
# TODO: restrict this to work with online StreamFeatureViews and validate the FeatureView type
try:
feature_view: FeatureView = self.get_stream_feature_view(
feature_view_name, allow_registry_cache=allow_registry_cache
)
except FeatureViewNotFoundException:
try:
feature_view = self.get_feature_view(
feature_view_name, allow_registry_cache=allow_registry_cache
)
except FeatureViewNotFoundException:
feature_view = self.get_label_view( # type: ignore[assignment]
feature_view_name, allow_registry_cache=allow_registry_cache
)
For a plain FeatureView — the common case — the first lookup cannot succeed. It
is issued, fails, raises, and is discarded, and only then does the lookup that
can succeed run. A LabelView pays two failed lookups before the third. The
number of registry lookups per call is a function of how far down the chain the
name happens to sit, and the first one is never the answer for anything except a
StreamFeatureView.
Every getter forwards allow_registry_cache as the registry's allow_cache
argument. On a registry deriving from CachingRegistry the failed attempts are
served from cached_registry_proto and cost little. On RemoteRegistry each is
a gRPC round-trip to the registry server, because that class keeps no
client-side cache and forwards allow_cache to the server as a request field
(filed separately as #6672).
The # TODO above the block records that the type dispatch here is unfinished.
Observed impact. This sits on a per-batch write path. In a backfill that
writes to the offline store in batches, every batch pays a guaranteed-miss
registry RPC before the real one — the exception is control flow across a
network hop. The work is proportional to batch count, and none of it can
succeed. allow_registry_cache=True, the default, does not avoid it on a remote
registry.
#4235 raised the same underlying asymmetry — three separate getters, and a
caller holding only a name cannot know which one to use — and asked for it to be
addressed in BaseRegistry. It was closed as completed and the fix shipped as
registry.get_any_feature_view (see below), but this call site was never
migrated to it — it still performs the try/except chain, and a third branch has
been added since.
Steps to reproduce
-
Configure a remote registry and apply a plain FeatureView with a batch
source:
project: demo
registry:
registry_type: remote
path: registry.example:80
-
Call store.write_to_offline_store("my_fv", df).
-
Observe two registry RPCs on the wire for one write: GetStreamFeatureView
(fails, FeatureViewNotFoundException) followed by GetFeatureView.
Either read the registry server's access log, or wrap
RemoteRegistry.stub and count calls.
-
Repeat the call in a loop, as a batched backfill does. The failed
GetStreamFeatureView recurs on every iteration, with
allow_registry_cache left at its True default.
Specifications
- Version: 0.62.0; verified present on 0.65.0 (latest release) and
master
- Platform: Linux, Python 3.12, Kubernetes; remote registry over gRPC, PostgreSQL offline store
- Subsystem:
feast.feature_store.FeatureStore.write_to_offline_store
Possible Solution
The API to fix this already exists. #4235's outcome shipped as
registry.get_any_feature_view, which resolves a name to a BaseFeatureView in
one lookup:
BaseRegistry.get_any_feature_view (abstract, base_registry.py:546)
CachingRegistry.get_any_feature_view (caching_registry.py:113) and
RemoteRegistry.get_any_feature_view (remote.py:387, RPC
GetAnyFeatureView, handler at registry_server.py:422)
proto_registry_utils.get_any_feature_view (proto_registry_utils.py:129)
searches feature_views, then stream_feature_views, then
on_demand_feature_views, then label_views — precisely the set the
try/except chain guesses at
feature_store.py already calls it in four places (lines 1067, 1078, 1090,
2985). write_to_offline_store was simply never migrated to it. Replacing the
chain with a single get_any_feature_view call collapses three potential
lookups into one and retires the # TODO.
Two details a reviewer will want, neither a blocker:
-
It returns BaseFeatureView, so the local annotation
feature_view: FeatureView needs an isinstance(fv, FeatureView) narrowing
to stay type-correct. That guard also gives the type validation the # TODO
asks for, and lets the # type: ignore[assignment] on the label branch go.
-
It is a behaviour change for one input. The current chain never resolves an
OnDemandFeatureView, so an ODFV name raises
FeatureViewNotFoundException today; via get_any_feature_view it would
resolve and then fail the has no batch_source check further down instead.
Arguably the better error, but it is a change.
Separately, and lower priority: a caller writing many batches for one feature
view holds the object already and can only hand back its name, so Feast
re-resolves per batch. An optional pre-resolved FeatureView parameter would let
batch writers resolve once per run. We currently sidestep this by calling
provider.ingest_df_to_offline_store directly, but that is not equivalent — it
takes a pyarrow.Table rather than a DataFrame and skips this method's
column-set validation and reorder_columns, so it is a workaround rather than a
recommendation.
Happy to open a PR for the get_any_feature_view migration if that looks right.
Expected Behavior
write_to_offline_storeshould resolve the feature view with one registry lookup,regardless of which feature view type the name refers to.
Current Behavior
FeatureStore.write_to_offline_storeresolves the feature view by trying eachgetter in turn and catching
FeatureViewNotFoundException(
sdk/python/feast/feature_store.py, lines 3650-3663 onmaster; the methodbegins at 3637):
For a plain
FeatureView— the common case — the first lookup cannot succeed. Itis issued, fails, raises, and is discarded, and only then does the lookup that
can succeed run. A
LabelViewpays two failed lookups before the third. Thenumber of registry lookups per call is a function of how far down the chain the
name happens to sit, and the first one is never the answer for anything except a
StreamFeatureView.Every getter forwards
allow_registry_cacheas the registry'sallow_cacheargument. On a registry deriving from
CachingRegistrythe failed attempts areserved from
cached_registry_protoand cost little. OnRemoteRegistryeach isa gRPC round-trip to the registry server, because that class keeps no
client-side cache and forwards
allow_cacheto the server as a request field(filed separately as #6672).
The
# TODOabove the block records that the type dispatch here is unfinished.Observed impact. This sits on a per-batch write path. In a backfill that
writes to the offline store in batches, every batch pays a guaranteed-miss
registry RPC before the real one — the exception is control flow across a
network hop. The work is proportional to batch count, and none of it can
succeed.
allow_registry_cache=True, the default, does not avoid it on a remoteregistry.
#4235 raised the same underlying asymmetry — three separate getters, and a
caller holding only a name cannot know which one to use — and asked for it to be
addressed in
BaseRegistry. It was closed as completed and the fix shipped asregistry.get_any_feature_view(see below), but this call site was nevermigrated to it — it still performs the try/except chain, and a third branch has
been added since.
Steps to reproduce
Configure a remote registry and apply a plain
FeatureViewwith a batchsource:
Call
store.write_to_offline_store("my_fv", df).Observe two registry RPCs on the wire for one write:
GetStreamFeatureView(fails,
FeatureViewNotFoundException) followed byGetFeatureView.Either read the registry server's access log, or wrap
RemoteRegistry.stuband count calls.Repeat the call in a loop, as a batched backfill does. The failed
GetStreamFeatureViewrecurs on every iteration, withallow_registry_cacheleft at itsTruedefault.Specifications
masterfeast.feature_store.FeatureStore.write_to_offline_storePossible Solution
The API to fix this already exists. #4235's outcome shipped as
registry.get_any_feature_view, which resolves a name to aBaseFeatureViewinone lookup:
BaseRegistry.get_any_feature_view(abstract,base_registry.py:546)CachingRegistry.get_any_feature_view(caching_registry.py:113) andRemoteRegistry.get_any_feature_view(remote.py:387, RPCGetAnyFeatureView, handler atregistry_server.py:422)proto_registry_utils.get_any_feature_view(proto_registry_utils.py:129)searches
feature_views, thenstream_feature_views, thenon_demand_feature_views, thenlabel_views— precisely the set thetry/except chain guesses at
feature_store.pyalready calls it in four places (lines 1067, 1078, 1090,2985).
write_to_offline_storewas simply never migrated to it. Replacing thechain with a single
get_any_feature_viewcall collapses three potentiallookups into one and retires the
# TODO.Two details a reviewer will want, neither a blocker:
It returns
BaseFeatureView, so the local annotationfeature_view: FeatureViewneeds anisinstance(fv, FeatureView)narrowingto stay type-correct. That guard also gives the type validation the
# TODOasks for, and lets the
# type: ignore[assignment]on the label branch go.It is a behaviour change for one input. The current chain never resolves an
OnDemandFeatureView, so an ODFV name raisesFeatureViewNotFoundExceptiontoday; viaget_any_feature_viewit wouldresolve and then fail the
has no batch_sourcecheck further down instead.Arguably the better error, but it is a change.
Separately, and lower priority: a caller writing many batches for one feature
view holds the object already and can only hand back its name, so Feast
re-resolves per batch. An optional pre-resolved
FeatureViewparameter would letbatch writers resolve once per run. We currently sidestep this by calling
provider.ingest_df_to_offline_storedirectly, but that is not equivalent — ittakes a
pyarrow.Tablerather than aDataFrameand skips this method'scolumn-set validation and
reorder_columns, so it is a workaround rather than arecommendation.
Happy to open a PR for the
get_any_feature_viewmigration if that looks right.