Skip to content

'feast apply' does not create online store tables in remote mode #6693

Description

@yuan1j

Expected Behavior

With online_store.type: remote + registry.registry_type: remote, a client-side feast apply should create the corresponding feature view tables in the server-side online store, so that feast materialize can load data successfully — i.e. the schema setup done by SqliteOnlineStore.update() during a local feast apply should also take effect on the feature-server side when feature views are applied through the remote registry.

Current Behavior

feast apply (remote) succeeds and registers the feature view, but no table is created server-side:

  • After feast apply, online_store.db does not exist (only registry_sqlite.db is present).
  • feast materialize reads offline data on the client, POSTs batches to the feature server's /write-to-online-store, and gets HTTP 500. The materialize command fails:
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=6566): Max retries exceeded with url: /write-to-online-store (Caused by ResponseError('too many 500 error responses'))

Feature server log shows the root cause:

File ".../feast/feature_store.py", line 3126, in write_to_online_store
    provider.ingest_df(feature_view, df)
File ".../feast/infra/passthrough_provider.py", line 197, in online_write_batch
    self.online_store.online_write_batch(config, table, data, progress)
File ".../feast/infra/online_stores/sqlite.py", line 220, in online_write_batch
    conn.execute(
sqlite3.OperationalError: no such table: test_project_driver_hourly_stats

After the failed materialize, online_store.db exists but is 0 bytes with zero tables.

Root cause (in source):

  • RemoteOnlineStore.update() is a no-op (pass): feast/infra/online_stores/remote.py L706-715. feast apply relies on OnlineStore.update() to create tables, so remote apply never creates them on the server-side online store.
  • feast serve does not call SqliteOnlineStore.update() for newly-applied feature views, so the table is never created server-side either.
  • SqliteOnlineStore.online_write_batch() only INSERTs (no CREATE TABLE): feast/infra/online_stores/sqlite.py L152-239. Table creation happens only in update() (L291-315), which is never invoked in this flow.

Net effect: the remote online store path is unusable for materialization unless the user manually runs feast apply locally on the feature-server repo, defeating the purpose of the remote architecture.

Steps to reproduce

Setup: a server repo (test_project/feature_repo) hosting registry + online store, and a client repo (test_demo) using remote registry + remote online store.

  1. Server repo test_project/feature_repo/feature_store.yaml:
project: test_project
registry:
  registry_type: sql
  path: sqlite:///data/registry_sqlite.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db
entity_key_serialization_version: 3
auth:
    type: no_auth
  1. Client repo test_demo/feature_store.yaml (remote registry + remote online store):
project: test_project
registry:
  registry_type: remote
  path: localhost:6570          # feast serve_registry (gRPC)
provider: local
online_store:
    type: remote
    path: http://localhost:6566  # feast serve (HTTP feature server)
entity_key_serialization_version: 3
auth:
    type: no_auth
  1. Client repo test_demo/feature_definitions.py (a single simple FeatureView):
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource, Project
from feast.types import Float32, Int64

project = Project(name="test_project", description="A project for driver statistics")
driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats_source = FileSource(
    name="driver_hourly_stats_source",
    path="data/driver_stats.parquet",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    ttl=timedelta(days=1),
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    online=True,
    source=driver_stats_source,
    version="latest",
)
  1. Start the servers from the server repo:
cd test_project/feature_repo
feast serve_registry --port 6570     # gRPC registry server
feast serve --port 6566              # HTTP feature server (separate shell)
  1. From the client repo, apply (remote):
cd test_demo
feast apply
# Applying changes for project test_project
# Deploying infrastructure for driver_hourly_stats
  1. Inspect the server-side online store — online_store.db does not exist (only registry_sqlite.db):
ls test_project/feature_repo/data
# registry_sqlite.db   (online_store.db is absent)
  1. From the client repo, materialize (remote):
cd test_demo
feast materialize 2021-04-12T00:00:00 2026-08-02T00:00:00
  1. Observe the failure: client fails with too many 500 error responses against /write-to-online-store; feature server logs sqlite3.OperationalError: no such table: test_project_driver_hourly_stats; online_store.db is left as a 0-byte file with no tables.

Specifications

  • Version: Feast SDK 0.64.0 (Python 3.12.10)
  • Platform: Windows 11 (10.0.26200), AMD64 — relevant code is platform-independent, expected to reproduce on Linux/macOS too
  • Subsystem: infra/online_stores/remote (RemoteOnlineStore), infra/online_stores/sqlite (SqliteOnlineStore), feature server (feast serve / write_to_online_store)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions