Skip to content

write_to_offline_store resolves the feature view by exception chain: a plain FeatureView always pays a guaranteed-miss GetStreamFeatureView RPC first, once per batch #6671

Description

@BigyaPradhan

Expected Behavior

write_to_offline_store should resolve the feature view with one registry lookup,
regardless of which feature view type the name refers to.

Current Behavior

FeatureStore.write_to_offline_store resolves the feature view by trying each
getter in turn and catching FeatureViewNotFoundException
(sdk/python/feast/feature_store.py, lines 3650-3663 on master; the method
begins at 3637):

# TODO: restrict this to work with online StreamFeatureViews and validate the FeatureView type
try:
    feature_view: FeatureView = self.get_stream_feature_view(
        feature_view_name, allow_registry_cache=allow_registry_cache
    )
except FeatureViewNotFoundException:
    try:
        feature_view = self.get_feature_view(
            feature_view_name, allow_registry_cache=allow_registry_cache
        )
    except FeatureViewNotFoundException:
        feature_view = self.get_label_view(  # type: ignore[assignment]
            feature_view_name, allow_registry_cache=allow_registry_cache
        )

For a plain FeatureView — the common case — the first lookup cannot succeed. It
is issued, fails, raises, and is discarded, and only then does the lookup that
can succeed run. A LabelView pays two failed lookups before the third. The
number of registry lookups per call is a function of how far down the chain the
name happens to sit, and the first one is never the answer for anything except a
StreamFeatureView.

Every getter forwards allow_registry_cache as the registry's allow_cache
argument. On a registry deriving from CachingRegistry the failed attempts are
served from cached_registry_proto and cost little. On RemoteRegistry each is
a gRPC round-trip to the registry server, because that class keeps no
client-side cache and forwards allow_cache to the server as a request field
(filed separately as #6672).

The # TODO above the block records that the type dispatch here is unfinished.

Observed impact. This sits on a per-batch write path. In a backfill that
writes to the offline store in batches, every batch pays a guaranteed-miss
registry RPC before the real one — the exception is control flow across a
network hop. The work is proportional to batch count, and none of it can
succeed. allow_registry_cache=True, the default, does not avoid it on a remote
registry.

#4235 raised the same underlying asymmetry — three separate getters, and a
caller holding only a name cannot know which one to use — and asked for it to be
addressed in BaseRegistry. It was closed as completed and the fix shipped as
registry.get_any_feature_view (see below), but this call site was never
migrated to it — it still performs the try/except chain, and a third branch has
been added since.

Steps to reproduce

  1. Configure a remote registry and apply a plain FeatureView with a batch
    source:

    project: demo
    registry:
      registry_type: remote
      path: registry.example:80
  2. Call store.write_to_offline_store("my_fv", df).

  3. Observe two registry RPCs on the wire for one write: GetStreamFeatureView
    (fails, FeatureViewNotFoundException) followed by GetFeatureView.
    Either read the registry server's access log, or wrap
    RemoteRegistry.stub and count calls.

  4. Repeat the call in a loop, as a batched backfill does. The failed
    GetStreamFeatureView recurs on every iteration, with
    allow_registry_cache left at its True default.

Specifications

  • Version: 0.62.0; verified present on 0.65.0 (latest release) and master
  • Platform: Linux, Python 3.12, Kubernetes; remote registry over gRPC, PostgreSQL offline store
  • Subsystem: feast.feature_store.FeatureStore.write_to_offline_store

Possible Solution

The API to fix this already exists. #4235's outcome shipped as
registry.get_any_feature_view, which resolves a name to a BaseFeatureView in
one lookup:

  • BaseRegistry.get_any_feature_view (abstract, base_registry.py:546)
  • CachingRegistry.get_any_feature_view (caching_registry.py:113) and
    RemoteRegistry.get_any_feature_view (remote.py:387, RPC
    GetAnyFeatureView, handler at registry_server.py:422)
  • proto_registry_utils.get_any_feature_view (proto_registry_utils.py:129)
    searches feature_views, then stream_feature_views, then
    on_demand_feature_views, then label_views — precisely the set the
    try/except chain guesses at

feature_store.py already calls it in four places (lines 1067, 1078, 1090,
2985). write_to_offline_store was simply never migrated to it. Replacing the
chain with a single get_any_feature_view call collapses three potential
lookups into one and retires the # TODO.

Two details a reviewer will want, neither a blocker:

  1. It returns BaseFeatureView, so the local annotation
    feature_view: FeatureView needs an isinstance(fv, FeatureView) narrowing
    to stay type-correct. That guard also gives the type validation the # TODO
    asks for, and lets the # type: ignore[assignment] on the label branch go.

  2. It is a behaviour change for one input. The current chain never resolves an
    OnDemandFeatureView, so an ODFV name raises
    FeatureViewNotFoundException today; via get_any_feature_view it would
    resolve and then fail the has no batch_source check further down instead.
    Arguably the better error, but it is a change.

Separately, and lower priority: a caller writing many batches for one feature
view holds the object already and can only hand back its name, so Feast
re-resolves per batch. An optional pre-resolved FeatureView parameter would let
batch writers resolve once per run. We currently sidestep this by calling
provider.ingest_df_to_offline_store directly, but that is not equivalent — it
takes a pyarrow.Table rather than a DataFrame and skips this method's
column-set validation and reorder_columns, so it is a workaround rather than a
recommendation.

Happy to open a PR for the get_any_feature_view migration if that looks right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions