Support Feast Map feature type for snowflake connection #5941
Description
Is your feature request related to a problem? Please describe.
There is feast data type called Map, added recently. Unfortunately, snowflake offline store doesn't support this type yet. (We don't have this type in snowflake_type_to_feast_value_type)
Describe the solution you'd like
Add possibility to define feast feature view with Map type for snowflake offline source with object/variant column
Additional context
Reproduce the problem.
- Define snowflake table
create or replace transient table ml.snowflake_table_example as
select current_date as date,
object_construct(
'feature1', 1,
'feature2', 'value2'
)::object as features_object
- Define feast objects
from datetime import timedelta
from feast import FeatureView, Field, SnowflakeSource
from feast.types import Map
snowflake_table_example_source = SnowflakeSource(
name="snowflake_table_example_source",
table="SNOWFLAKE_TABLE_EXAMPLE",
timestamp_field="DATE",
)
snowflake_table_example_fv = FeatureView(
name="snowflake_table_example_fv",
entities=[],
ttl=timedelta(weeks=52 * 10),
schema=[
Field(name="FEATURES_OBJECT", dtype=Map),
],
source=snowflake_table_example_source,
)
- When run
feast apply, getting this error
Applying changes for project gs
Traceback (most recent call last):
File "/usr/local/bin/feast", line 8, in <module>
sys.exit(cli())
^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/feast/cli/cli.py", line 302, in apply_total_command
apply_total(
File "/usr/local/lib/python3.12/site-packages/feast/repo_operations.py", line 451, in apply_total
apply_total_with_repo_instance(
File "/usr/local/lib/python3.12/site-packages/feast/repo_operations.py", line 351, in apply_total_with_repo_instance
provider.validate_data_source(store.config, data_source)
File "/usr/local/lib/python3.12/site-packages/feast/infra/passthrough_provider.py", line 555, in validate_data_source
self.offline_store.validate_data_source(config=config, data_source=data_source)
File "/usr/local/lib/python3.12/site-packages/feast/infra/offline_stores/offline_store.py", line 462, in validate_data_source
data_source.validate(config=config)
File "/usr/local/lib/python3.12/site-packages/feast/infra/offline_stores/snowflake_source.py", line 192, in validate
self.get_table_column_names_and_types(config)
File "/usr/local/lib/python3.12/site-packages/feast/infra/offline_stores/snowflake_source.py", line 294, in get_table_column_names_and_types
raise NotImplementedError(
NotImplementedError: The datatype of column FEATURES_OBJECT is of type OBJECT in datasource SELECT * FROM "SNOWFLAKE_TABLE_EXAMPLE" LIMIT 5. This type is not supported. Try converting to VARCHAR.
As alternative, we can add Map type as varchar snowflake type, where stored JSON-string.
Currently, if we define snowflake table with json-string column
create or replace transient table ml.snowflake_table_example as
select current_date as date,
object_construct(
'feature1', 1,
'feature2', 'value2'
)::string as features_object
And run feast materialise, getting this error
snowflake_table_example_fv:
0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/feast", line 8, in <module>
sys.exit(cli())
^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/feast/cli/cli.py", line 393, in materialize_command
store.materialize(
File "/usr/local/lib/python3.12/site-packages/feast/feature_store.py", line 1684, in materialize
provider.materialize_single_feature_view(
File "/usr/local/lib/python3.12/site-packages/feast/infra/passthrough_provider.py", line 456, in materialize_single_feature_view
raise e
File "/usr/local/lib/python3.12/site-packages/feast/infra/compute_engines/snowflake/snowflake_engine.py", line 304, in _materialize_one
fv_to_proto_sql = self.generate_snowflake_materialization_query(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/feast/infra/compute_engines/snowflake/snowflake_engine.py", line 366, in generate_snowflake_materialization_query
feature_sql = _convert_value_name_to_snowflake_udf(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/feast/type_map.py", line 872, in _convert_value_name_to_snowflake_udf
return name_map[value_name].upper()
~~~~~~~~^^^^^^^^^^^^
KeyError: 'MAP'