-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Closed
Labels
Description
Expected Behavior
When deployed with the Operator, Feast should be fully functional.
Current Behavior
Several methods failing... e.g.
$ feast apply
No project found in the repository. Using project name sample defined in feature_store.yaml
Applying changes for project sample
/feast/sdk/python/feast/feature_store.py:575: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval
warnings.warn(
12/05/2024 02:35:01 PM feast.errors WARNING: Invalid error detail: : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/usr/local/bin/feast", line 8, in <module>
sys.exit(cli())
^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/cli.py", line 760, in apply_total_command
apply_total(repo_config, repo, skip_source_validation)
File "/feast/sdk/python/feast/repo_operations.py", line 404, in apply_total
apply_total_with_repo_instance(
File "/feast/sdk/python/feast/repo_operations.py", line 351, in apply_total_with_repo_instance
store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
File "/feast/sdk/python/feast/feature_store.py", line 933, in apply
self._make_inferences(
File "/feast/sdk/python/feast/feature_store.py", line 613, in _make_inferences
update_feature_views_with_inferred_features_and_entities(
File "/feast/sdk/python/feast/inference.py", line 180, in update_feature_views_with_inferred_features_and_entities
_infer_features_and_entities(
File "/feast/sdk/python/feast/inference.py", line 230, in _infer_features_and_entities
provider.get_table_column_names_and_types_from_data_source(
File "/feast/sdk/python/feast/infra/passthrough_provider.py", line 524, in get_table_column_names_and_types_from_data_source
return self.offline_store.get_table_column_names_and_types_from_data_source(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 420, in get_table_column_names_and_types_from_data_source
table = _send_retrieve_remote(
^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 494, in _send_retrieve_remote
return _call_get(client, command_descriptor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 503, in _call_get
reader = client.do_get(ticket)
^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/arrow_error_handler.py", line 20, in wrapper
raise e
File "/feast/sdk/python/feast/arrow_error_handler.py", line 15, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 53, in do_get
return super().do_get(ticket, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_flight.pyx", line 1633, in pyarrow._flight.FlightClient.do_get
File "pyarrow/_flight.pyx", line 58, in pyarrow._flight.check_flight_status
pyarrow._flight.FlightServerError: FlightServerBase.do_get must return a FlightDataStream. Detail: Python exception: Traceback (most recent call last):
File "pyarrow/_flight.pyx", line 2185, in pyarrow._flight._do_get
TypeError: FlightServerBase.do_get must return a FlightDataStream
. gRPC client debug context: UNKNOWN:Error received from peer ipv4:172.30.141.142:80 {grpc_message:"FlightServerBase.do_get must return a FlightDataStream. Detail: Python exception: Traceback (most recent call last):\n File \"pyarrow/_flight.pyx\", line 2185, in pyarrow._flight._do_get\nTypeError: FlightServerBase.do_get must return a FlightDataStream\n", grpc_status:2, created_time:"2024-12-05T14:35:01.244475681+00:00"}. Client context: IOError: Server never sent a data message. Detail: Internal $ python3 test_workflow.py
--- Run feast apply to setup feature store on Postgres ---
No project found in the repository. Using project name sample defined in feature_store.yaml
Applying changes for project sample
/feast/sdk/python/feast/feature_store.py:575: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval
warnings.warn(
Deploying infrastructure for driver_hourly_stats_fresh
Deploying infrastructure for driver_hourly_stats--- Historical features for training ---
driver_id event_timestamp label_driver_reported_satisfaction val_to_add val_to_add_2 conv_rate acc_rate avg_daily_trips
0 1001 2021-04-12 10:59:42 1 1 10 0.654609 0.089119 56
1 1002 2021-04-12 08:12:10 5 2 20 0.628438 0.045928 642
2 1003 2021-04-12 16:40:26 3 3 30 0.597835 0.370411 682--- Historical features for batch scoring ---
driver_id event_timestamp label_driver_reported_satisfaction val_to_add val_to_add_2 conv_rate acc_rate avg_daily_trips
0 1001 2024-12-05 14:45:02.856623 1 1 10 0.258632 0.935114 59
1 1002 2024-12-05 14:45:02.856623 5 2 20 0.294726 0.869369 341
2 1003 2024-12-05 14:45:02.856623 3 3 30 0.759920 0.541704 69--- Load features into online store ---
Materializing 2 feature views to 2024-12-05 14:45:03+00:00 into the remote online store.driver_hourly_stats from 2024-12-05 14:44:04+00:00 to 2024-12-05 14:45:03+00:00:
0it [00:00, ?it/s]
driver_hourly_stats_fresh from 2024-12-05 14:44:04+00:00 to 2024-12-05 14:45:03+00:00:
0it [00:00, ?it/s]--- Online features ---
Traceback (most recent call last):
File "/feast-init/sample/feature_repo/test_workflow.py", line 131, in <module>
run_demo()
File "/feast-init/sample/feature_repo/test_workflow.py", line 26, in run_demo
fetch_online_features(store)
File "/feast-init/sample/feature_repo/test_workflow.py", line 122, in fetch_online_features
returned_features = store.get_online_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/feature_store.py", line 1696, in get_online_features
return provider.get_online_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/passthrough_provider.py", line 249, in get_online_features
return self.online_store.get_online_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/online_stores/online_store.py", line 179, in get_online_features
) = utils._prepare_entities_to_read_from_online_store(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/utils.py", line 1036, in _prepare_entities_to_read_from_online_store
) = _get_online_request_context(registry, project, features, full_feature_names)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/utils.py", line 959, in _get_online_request_context
) = _get_feature_views_to_use(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/utils.py", line 907, in _get_feature_views_to_use
fv = registry.get_any_feature_view(name, project, allow_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/feast/sdk/python/feast/infra/registry/remote.py", line 290, in get_any_feature_view Specifications
- Version: 0.42.0
- Platform: Kubernetes
- Subsystem: Feast deployed via Go Operator
Possible Solution
Move away from using remote provider for service interaction.