Unable to feast apply when using SQL Registry #3000
Description
Expected Behavior
Using a SQL registry, we should be able to run feast apply more than once for a given project, so that feast can identify the diff between the existing registry and project.
Current Behavior
When using a SQL registry, running feast apply for the first time works. However, when running it again, it returns the following error:
Traceback (most recent call last):
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/bin/feast", line 8, in <module>
sys.exit(cli())
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/cli.py", line 492, in apply_total_command
apply_total(repo_config, repo, skip_source_validation)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 274, in wrapper
return func(*args, **kwargs)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py", line 305, in apply_total
apply_total_with_repo_instance(
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py", line 265, in apply_total_with_repo_instance
registry_diff, infra_diff, new_infra = store.plan(repo)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 285, in wrapper
raise exc.with_traceback(traceback)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 274, in wrapper
return func(*args, **kwargs)
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/feature_store.py", line 708, in plan
current_infra_proto = self._registry.proto().infra.__deepcopy__()
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 666, in proto
r.infra.CopyFrom(self.get_infra(project).to_proto())
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 558, in get_infra
return self._get_object(
File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 774, in _get_object
raise not_found_exception(name, project)
TypeError: 'NoneType' object is not callable
This error is raised because of what looks like a bug in SqlRegistry.get_infra(). Indeed, SqlRegistry.get_infra() calls SqlRegistry._get_object() but passing None for the not_found_exception argument.
feast/infra/registry_stores/sql.py
...
def get_infra(self, project: str, allow_cache: bool = False) -> Infra:
return self._get_object(
managed_infra,
"infra_obj",
project,
InfraProto,
Infra,
"infra_name",
"infra_proto",
>>>> None, <<<<
)
...
def _get_object(
self,
table,
name,
project,
proto_class,
python_class,
id_field_name,
proto_field_name,
>>> not_found_exception <<<,
):
self._maybe_init_project_metadata(project)
with self.engine.connect() as conn:
stmt = select(table).where(
getattr(table.c, id_field_name) == name, table.c.project_id == project
)
row = conn.execute(stmt).first()
if row:
_proto = proto_class.FromString(row[proto_field_name])
return python_class.from_proto(_proto)
>>> raise not_found_exception(name, project) <<<Since no rows are returned for the managed_infra table, _get_object attempts to raise an error with the not_found_exception callable provided, but since None was provided, it is unable to raise the error correctly.
This seems to be hiding the actual error, which is that no rows are returned from the managed_infra table, which I can confirm is indeed empty in the SQL backend. I've provided my feature_store.yaml configuration below -- is the expectation that something should be created in the managed_infra table?
Steps to reproduce
feature_store.yaml
project: feature_store
provider: local
registry:
registry_type: sql
path: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DATABASE}
offline_store:
type: snowflake.offline
account: ${SNOWFLAKE_ACCOUNT}
user: ${SNOWFLAKE_USER}
password: ${SNOWFLAKE_PASSWORD}
role: ${SNOWFLAKE_ROLE}
warehouse: ${SNOWFLAKE_WAREHOUSE}
database: ${SNOWFLAKE_DATABASE}
schema: ${SNOWFLAKE_SCHEMA}
online_store:
type: redis
redis_type: redis
connection_string: "${REDIS_HOST}:${REDIS_PORT},password=${REDIS_PASSWORD}"Run the following commands:
feast apply # works
feast apply # raises the error above
Specifications
- Version: 0.22.1
- Platform: macOS
- Subsystem: 12.4