Skip to content

Commit 037c4cd

Browse files
AdityaPatil22AdityaPatil22
authored
feat: Added SQL registry schema_mode and registry create command (#6704)
1 parent b7ae488 commit 037c4cd

7 files changed

Lines changed: 221 additions & 7 deletions

File tree

‎docs/reference/feast-cli-commands.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Commands:
2828
materialize Run a (non-incremental) materialization job to...
2929
materialize-incremental Run an incremental materialization job to ingest...
3030
permissions Access permissions
31+
registry Manage the feature registry
3132
registry-dump Print contents of the metadata registry
3233
teardown Tear down deployed feature store infrastructure
3334
version Display Feast SDK version
@@ -483,6 +484,18 @@ reader driver_hourly_stats_fresh FeatureView DESCRIBE
483484
```
484485

485486

487+
## Registry
488+
489+
### create-schema
490+
491+
Pre-create the SQL registry schema so the application does not need DDL privileges at runtime. Use this with `schema_mode: verify` or `schema_mode: skip` in your `feature_store.yaml`.
492+
493+
```text
494+
feast registry create-schema
495+
```
496+
497+
This command only applies to SQL-based registries (`registry_type: sql`). It is safe to run multiple times — existing tables are not modified.
498+
486499
## Teardown
487500

488501
Tear down deployed feature store infrastructure

‎docs/reference/feature-store-yaml.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ registry:
6666
|-------|------|---------|-------------|
6767
| `registry_type` | string | `file` | Registry backend (`file`, `sql`, etc.) |
6868
| `path` | string | — | Connection string or file path |
69+
| `schema_mode` | string | `auto` | SQL registry only. `auto`: create tables on startup; `verify`: check tables exist, error if missing; `skip`: no DDL or verification. See [SQL Registry docs](registries/sql.md#schema-management-schema_mode). |
6970
| `mcp.enabled` | bool | `false` | Enable MCP (Model Context Protocol) on the REST registry server |
7071

7172
When `registry.mcp.enabled` is `true`, the REST registry server exposes registry

‎docs/reference/registries/sql.md‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,36 @@ docker build \
8080
If you are running Feast in Kubernetes, set the `image.repository` and
8181
`imagePullSecrets` Helm values accordingly to utilize your custom image.
8282

83+
## Schema management (`schema_mode`)
84+
85+
By default, the SQL registry creates its tables on every startup (`schema_mode: auto`). In production environments where the application should not have DDL privileges, you can pre-create the schema and configure the registry to only verify it:
86+
87+
```yaml
88+
registry:
89+
registry_type: sql
90+
path: postgresql://db:5432/feast
91+
schema_mode: verify # or "skip"
92+
```
93+
94+
| Value | Behavior |
95+
|---|---|
96+
| `auto` (default) | Creates tables if they don't exist. Current behavior, no breaking change. |
97+
| `verify` | Skips DDL. Checks that all expected tables exist on startup; raises an error listing missing tables if any are absent. When a separate `read_path` is configured, the read replica is also verified — a lagging replica (e.g. mid-migration) will block startup. Note: this is a table-level check only — it does not verify individual columns. A schema created by an older Feast version (missing newer columns) will pass verification but may fail at query time. |
98+
| `skip` | Skips both creation and verification. Use when schema is managed entirely outside Feast (e.g. by a migration tool). |
99+
100+
### Pre-creating the schema
101+
102+
When using `verify` or `skip` mode, run the following CLI command with a user that has DDL privileges to create the schema before starting the application:
103+
104+
```shell
105+
feast registry create-schema
106+
```
107+
108+
This reads `feature_store.yaml`, connects to the configured database, and creates all required tables. It is safe to run multiple times — existing tables are not modified.
109+
83110
There are some things to note about how the SQL registry works:
84-
- Once instantiated, the Registry ensures the tables needed to store data exist, and creates them if they do not.
85-
- Upon tearing down the feast project, the registry ensures that the tables are dropped from the database.
111+
- When `schema_mode` is `auto` (the default), the Registry ensures the tables needed to store data exist, and creates them if they do not.
112+
- Upon tearing down the feast project, the registry deletes all rows from the registry tables (it does not drop the tables themselves). This runs regardless of `schema_mode` and requires only DML (`DELETE`) privileges, not DDL.
86113
- The schema for how data is laid out in tables can be found in the table definitions in [`sdk/python/feast/infra/registry/sql.py`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/registry/sql.py). It is intentionally simple, storing the serialized protobuf versions of each Feast object keyed by its name.
87114

88115
## MySQL: serialized-proto columns use `LONGBLOB`

‎sdk/python/feast/cli/cli.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
from feast.cli.on_demand_feature_views import on_demand_feature_views_cmd
4141
from feast.cli.permissions import feast_permissions_cmd
4242
from feast.cli.projects import projects_cmd
43+
from feast.cli.registry import registry_cmd
4344
from feast.cli.saved_datasets import saved_datasets_cmd
4445
from feast.cli.serve import (
4546
serve_command,
@@ -644,6 +645,7 @@ def demo_notebooks_command(ctx: click.Context, output_dir: str, overwrite: bool)
644645
cli.add_command(on_demand_feature_views_cmd)
645646
cli.add_command(feast_permissions_cmd)
646647
cli.add_command(projects_cmd)
648+
cli.add_command(registry_cmd)
647649
cli.add_command(saved_datasets_cmd)
648650
cli.add_command(stream_feature_views_cmd)
649651
cli.add_command(label_views_cmd)

‎sdk/python/feast/cli/registry.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import click
2+
from sqlalchemy import create_engine
3+
4+
from feast.infra.registry.sql import SqlRegistryConfig, metadata
5+
from feast.repo_config import load_repo_config
6+
from feast.repo_operations import cli_check_repo
7+
8+
9+
@click.group(name="registry")
10+
def registry_cmd() -> None:
11+
"""
12+
Manage the feature registry
13+
"""
14+
pass
15+
16+
17+
@registry_cmd.command("create-schema")
18+
@click.pass_context
19+
def registry_create_schema(ctx: click.Context) -> None:
20+
"""
21+
Pre-create the SQL registry schema.
22+
23+
Use this when schema_mode is set to 'verify' or 'skip' so the application
24+
does not need DDL privileges at runtime.
25+
"""
26+
repo = ctx.obj["CHDIR"]
27+
fs_yaml_file = ctx.obj["FS_YAML_FILE"]
28+
cli_check_repo(repo, fs_yaml_file)
29+
repo_config = load_repo_config(repo, fs_yaml_file)
30+
31+
if repo_config is None:
32+
raise click.ClickException("Could not load feature_store.yaml")
33+
34+
registry_config = repo_config.registry
35+
if not isinstance(registry_config, SqlRegistryConfig):
36+
raise click.ClickException(
37+
"This command only applies to SQL-based registries "
38+
f"(registry_type='sql'). Current type: '{registry_config.registry_type}'"
39+
)
40+
41+
engine = create_engine(
42+
registry_config.path, **registry_config.sqlalchemy_config_kwargs
43+
)
44+
metadata.create_all(engine)
45+
engine.dispose()
46+
click.echo("SQL registry schema created successfully.")

‎sdk/python/feast/infra/registry/sql.py‎

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
from datetime import datetime, timezone
55
from enum import Enum
66
from pathlib import Path
7-
from typing import Any, Callable, Dict, List, Optional, Union, cast
7+
from typing import Any, Callable, Dict, List, Literal, Optional, Union, cast
88

99
from pydantic import StrictInt, StrictStr, field_validator
1010
from sqlalchemy import ( # type: ignore
@@ -26,6 +26,9 @@
2626
text,
2727
update,
2828
)
29+
from sqlalchemy import (
30+
inspect as sa_inspect,
31+
)
2932
from sqlalchemy.dialects import mysql
3033
from sqlalchemy.engine import Engine
3134
from sqlalchemy.exc import IntegrityError
@@ -306,6 +309,12 @@ class SqlRegistryConfig(RegistryConfig):
306309
thread_pool_executor_worker_count: StrictInt = 0
307310
""" int: Number of worker threads to use for asynchronous caching in SQL Registry. If set to 0, it doesn't use ThreadPoolExecutor. """
308311

312+
schema_mode: Literal["auto", "verify", "skip"] = "auto"
313+
""" str: Controls schema creation on startup.
314+
'auto' (default) — creates tables if they don't exist (current behavior).
315+
'verify' — skips DDL; checks that all expected tables exist and raises an error if any are missing.
316+
'skip' — skips both creation and verification. """
317+
309318
@field_validator("read_path")
310319
def validate_read_path(cls, read_path: Optional[str]) -> Optional[str]:
311320
# Mirror `RegistryConfig.validate_path`: a bare `postgresql://` read_path
@@ -316,6 +325,16 @@ def validate_read_path(cls, read_path: Optional[str]) -> Optional[str]:
316325
return read_path
317326

318327

328+
class FeastRegistrySchemaError(Exception):
329+
def __init__(self, missing_tables: List[str]) -> None:
330+
tables = ", ".join(missing_tables)
331+
super().__init__(
332+
f"SQL registry schema is incomplete — missing tables: {tables}. "
333+
"Run 'feast registry create-schema' to create them, "
334+
"or set schema_mode='auto' to create tables on startup."
335+
)
336+
337+
319338
class SqlRegistry(CachingRegistry):
320339
def __init__(
321340
self,
@@ -339,7 +358,12 @@ def __init__(
339358
)
340359
else:
341360
self.read_engine = self.write_engine
342-
metadata.create_all(self.write_engine)
361+
if registry_config.schema_mode == "auto":
362+
metadata.create_all(self.write_engine)
363+
elif registry_config.schema_mode == "verify":
364+
self._verify_schema(self.write_engine)
365+
if self.read_engine is not self.write_engine:
366+
self._verify_schema(self.read_engine)
343367
self._warn_if_narrow_blob_columns(self.write_engine)
344368
if self.read_engine is not self.write_engine:
345369
# A read replica can be on a different schema version (e.g. mid
@@ -357,13 +381,25 @@ def __init__(
357381
cache_ttl_seconds=registry_config.cache_ttl_seconds,
358382
cache_mode=registry_config.cache_mode,
359383
)
360-
# Sync feast_metadata to projects table
361-
# when purge_feast_metadata is set to True, Delete data from
362-
# feast_metadata table and list_project_metadata will not return any data
363384
self._sync_feast_metadata_to_projects_table()
364385
if not self.purge_feast_metadata:
365386
self._maybe_init_project_metadata(project)
366387

388+
@staticmethod
389+
def _verify_schema(engine: Engine, registry_metadata: MetaData = metadata) -> None:
390+
"""Verify that all expected registry tables exist in the database.
391+
392+
Raises ``FeastRegistrySchemaError`` listing missing tables and
393+
suggesting ``feast registry create-schema``.
394+
"""
395+
expected = set(registry_metadata.tables.keys())
396+
actual = set(
397+
sa_inspect(engine).get_table_names(schema=registry_metadata.schema)
398+
)
399+
missing = expected - actual
400+
if missing:
401+
raise FeastRegistrySchemaError(sorted(missing))
402+
367403
@staticmethod
368404
def _warn_if_narrow_blob_columns(
369405
engine: Engine, registry_metadata: MetaData = metadata

‎sdk/python/tests/unit/infra/registry/test_sql_registry.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,15 @@
2929
from feast.feature_view import FeatureView
3030
from feast.infra.offline_stores.file_source import FileSource
3131
from feast.infra.registry.sql import (
32+
FeastRegistrySchemaError,
3233
ProtoBytes,
3334
SqlRegistry,
3435
SqlRegistryConfig,
3536
feature_views,
3637
)
38+
from feast.infra.registry.sql import (
39+
metadata as registry_metadata,
40+
)
3741
from feast.protos.feast.core.Transformation_pb2 import (
3842
FeatureTransformationV2,
3943
UserDefinedFunctionV2,
@@ -585,3 +589,88 @@ def test_list_feature_views_updated_since_naive_treated_as_utc(sqlite_registry):
585589
"test_project", tags=None, updated_since=past_aware
586590
)
587591
assert len(result) == len(result_aware)
592+
593+
594+
class TestSchemaMode:
595+
def test_schema_mode_auto_creates_tables(self, tmp_path):
596+
"""Default schema_mode='auto' creates tables on init (existing behavior)."""
597+
db_file = tmp_path / "auto.db"
598+
config = SqlRegistryConfig(
599+
registry_type="sql",
600+
path=f"sqlite:///{db_file}",
601+
schema_mode="auto",
602+
)
603+
registry = SqlRegistry(config, "test_project", None)
604+
from sqlalchemy import create_engine, inspect
605+
606+
engine = create_engine(f"sqlite:///{db_file}")
607+
tables = set(inspect(engine).get_table_names())
608+
expected = set(registry_metadata.tables.keys())
609+
assert expected.issubset(tables)
610+
engine.dispose()
611+
registry.teardown()
612+
613+
def test_schema_mode_verify_raises_when_tables_missing(self, tmp_path):
614+
"""schema_mode='verify' raises FeastRegistrySchemaError on empty database."""
615+
db_file = tmp_path / "verify_empty.db"
616+
config = SqlRegistryConfig(
617+
registry_type="sql",
618+
path=f"sqlite:///{db_file}",
619+
schema_mode="verify",
620+
)
621+
with pytest.raises(FeastRegistrySchemaError, match="missing tables"):
622+
SqlRegistry(config, "test_project", None)
623+
624+
def test_schema_mode_verify_passes_when_tables_exist(self, tmp_path):
625+
"""schema_mode='verify' succeeds when schema was pre-created."""
626+
db_file = tmp_path / "verify_ok.db"
627+
db_url = f"sqlite:///{db_file}"
628+
from sqlalchemy import create_engine
629+
630+
engine = create_engine(db_url)
631+
registry_metadata.create_all(engine)
632+
engine.dispose()
633+
634+
config = SqlRegistryConfig(
635+
registry_type="sql",
636+
path=db_url,
637+
schema_mode="verify",
638+
)
639+
registry = SqlRegistry(config, "test_project", None)
640+
registry.teardown()
641+
642+
def test_schema_mode_skip_does_not_run_ddl(self, tmp_path):
643+
"""schema_mode='skip' calls neither create_all nor _verify_schema."""
644+
from unittest.mock import patch
645+
646+
db_file = tmp_path / "skip.db"
647+
db_url = f"sqlite:///{db_file}"
648+
from sqlalchemy import create_engine
649+
650+
engine = create_engine(db_url)
651+
registry_metadata.create_all(engine)
652+
engine.dispose()
653+
654+
with (
655+
patch.object(registry_metadata, "create_all") as mock_create,
656+
patch.object(SqlRegistry, "_verify_schema") as mock_verify,
657+
):
658+
config = SqlRegistryConfig(
659+
registry_type="sql",
660+
path=db_url,
661+
schema_mode="skip",
662+
)
663+
SqlRegistry(config, "test_project", None)
664+
mock_create.assert_not_called()
665+
mock_verify.assert_not_called()
666+
667+
def test_schema_mode_invalid_value_rejected(self):
668+
"""schema_mode only accepts 'auto', 'verify', 'skip'."""
669+
from pydantic import ValidationError
670+
671+
with pytest.raises(ValidationError):
672+
SqlRegistryConfig(
673+
registry_type="sql",
674+
path="sqlite:///dummy.db",
675+
schema_mode="bogus",
676+
)

0 commit comments

Comments
 (0)