Added milvus write batch functionality #17

Manisha4 · 2023-08-16T16:30:09Z

What this PR does / why we need it:
Added Write batch functionality for Milvus

Which issue(s) this PR fixes:

Fixes #

michaelbackes · 2023-08-16T18:45:40Z

sdk/python/feast/expediagroup/vectordb/milvus_online_store.py

+                collection_to_load_data.insert(entities)
+                #  The flush call will seal any remaining segments and send them for indexing
+                collection_to_load_data.flush()
+            except Exception as e:


I feel that we shouldn't put a try / except around the entire code and rather catch errors more targeted where they actually occur. We can address it later, but if you see ways to do it now please do so.

michaelbackes · 2023-08-16T19:29:17Z

sdk/python/feast/expediagroup/vectordb/milvus_online_store.py

                    if collection_available:
                        logger.info(f"Collection {table_to_keep.name} already exists.")
                    else:
+                        # TODO: Enable dynamic schema option


what does this mean?

Milvus allows arbitrary additional fields including JSON documents with a enable_dynamic_field=True property in the CollectionSchema.
It would make adding new fields to a collection very easy, but I don't know if that really works the way Feast does. I believe if the FeatureView schema would change then the old collection would be flagged for deletion and a new one would be created.

thanks for the explanation. Probably better to remove the to-do for now and have a spike in the future if that functionality is needed

Sure, I had that note in to think about how we could potentially incorporate it but a spike ticket seems fine.

michaelbackes · 2023-08-16T19:35:04Z

sdk/python/feast/expediagroup/vectordb/milvus_online_store.py

-        )
+        with MilvusConnectionManager(config.online_store):
+            try:
+                entities = self._format_data_for_milvus(data)


let's use a different term like "rows" instead so that this doesn't get confused with the Entity type

michaelbackes · 2023-08-16T19:41:02Z

sdk/python/feast/expediagroup/vectordb/milvus_online_store.py

+                    final_float_list = np.array(float_list)
+                    feature.append(final_float_list)
+                # TODO: Test out binary lists
+                if val_type == "biinary_list_val":


typo: should be binary_list_val

I think we could simplify this:

val_type = val.WhichOneof("val")
value = getattr(val, val_type).val
if val_type == "float_list_val":
value = np.array(value)
feature.append(value)

what do you think?

This only works for floats

value = getattr(val, val_type).val

else

for non floats
value = getattr(val, val_type)

Maybe I am not understanding it correctly but I don't feel like this approach works

michaelbackes · 2023-08-16T22:40:09Z

sdk/python/tests/expediagroup/test_milvus_online_store.py

+                ),
+                {
+                    "avg_orders_day": ValueProto(
+                        float_list_val=FloatList(val=[1, 2, 3, 4, 5])


minor: values are not floats

reverting yield commit

This reverts commit 97c4789.

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

# [0.49.0](feast-dev/feast@v0.48.0...v0.49.0) (2025-04-29) ### Bug Fixes * Adding brackets to unit tests ([c46fea3](feast-dev@c46fea3)) * Adding logic back for a step ([2bb240b](feast-dev@2bb240b)) * Adjustment for unit test action ([a6f78ae](feast-dev@a6f78ae)) * Allow get_historical_features with only On Demand Feature View ([feast-dev#5256](feast-dev#5256)) ([0752795](feast-dev@0752795)) * CI adjustment ([3850643](feast-dev@3850643)) * Embed Query configuration breaks when switching between DataFrame and SQL ([feast-dev#5257](feast-dev#5257)) ([32375a5](feast-dev@32375a5)) * Fix for proto issue in utils ([1b291b2](feast-dev@1b291b2)) * Fix milvus online_read ([feast-dev#5233](feast-dev#5233)) ([4b91f26](feast-dev@4b91f26)) * Fix tests ([431d9b8](feast-dev@431d9b8)) * Fixed Permissions object parameter in example ([feast-dev#5259](feast-dev#5259)) ([045c100](feast-dev@045c100)) * Java CI [#12](feast-dev#12) ([d7e44ac](feast-dev@d7e44ac)) * Java PR [#15](feast-dev#15) ([a5da3bb](feast-dev@a5da3bb)) * Java PR [#16](feast-dev#16) ([e0320fe](feast-dev@e0320fe)) * Java PR [#17](feast-dev#17) ([49da810](feast-dev@49da810)) * Materialization logs ([feast-dev#5243](feast-dev#5243)) ([4aa2f49](feast-dev@4aa2f49)) * Moving to custom github action for checking skip tests ([caf312e](feast-dev@caf312e)) * Operator - remove default replicas setting from Feast Deployment ([feast-dev#5294](feast-dev#5294)) ([e416d01](feast-dev@e416d01)) * Patch java pr [#14](feast-dev#14) ([592526c](feast-dev@592526c)) * Patch update for test ([a3e8967](feast-dev@a3e8967)) * Remove conditional from steps ([995307f](feast-dev@995307f)) * Remove misleading HTTP prefix from gRPC endpoints in logs and doc ([feast-dev#5280](feast-dev#5280)) ([0ee3a1e](feast-dev@0ee3a1e)) * removing id ([268ade2](feast-dev@268ade2)) * Renaming workflow file ([5f46279](feast-dev@5f46279)) * Resolve `no pq wrapper` import issue ([feast-dev#5240](feast-dev#5240)) ([d5906f1](feast-dev@d5906f1)) * Update actions to remove check skip tests ([feast-dev#5275](feast-dev#5275)) ([b976f27](feast-dev@b976f27)) * Update docling demo ([446efea](feast-dev@446efea)) * Update java pr [#13](feast-dev#13) ([fda7db7](feast-dev@fda7db7)) * Update java_pr ([fa138f4](feast-dev@fa138f4)) * Update repo_config.py ([6a59815](feast-dev@6a59815)) * Update unit tests workflow ([06486a0](feast-dev@06486a0)) * Updated docs for docling demo ([768e6cc](feast-dev@768e6cc)) * Updating action for unit tests ([0996c28](feast-dev@0996c28)) * Updating github actions to filter at job level ([0a09622](feast-dev@0a09622)) * Updating Java CI ([c7c3a3c](feast-dev@c7c3a3c)) * Updating java pr to skip tests ([e997dd9](feast-dev@e997dd9)) * Updating workflows ([c66bcd2](feast-dev@c66bcd2)) ### Features * Add date_partition_column_format for spark source ([feast-dev#5273](feast-dev#5273)) ([7a61d6f](feast-dev@7a61d6f)) * Add Milvus tutorial with Feast integration ([feast-dev#5292](feast-dev#5292)) ([a1388a5](feast-dev@a1388a5)) * Add pgvector tutorial with PostgreSQL integration ([feast-dev#5290](feast-dev#5290)) ([bb1cbea](feast-dev@bb1cbea)) * Add ReactFlow visualization for Feast registry metadata ([feast-dev#5297](feast-dev#5297)) ([9768970](feast-dev@9768970)) * Add retrieve online documents v2 method into pgvector ([feast-dev#5253](feast-dev#5253)) ([6770ee6](feast-dev@6770ee6)) * Compute Engine Initial Implementation ([feast-dev#5223](feast-dev#5223)) ([64bdafd](feast-dev@64bdafd)) * Enable write node for compute engine ([feast-dev#5287](feast-dev#5287)) ([f9baf97](feast-dev@f9baf97)) * Local compute engine ([feast-dev#5278](feast-dev#5278)) ([8e06dfe](feast-dev@8e06dfe)) * Make transform on writes configurable for ingestion ([feast-dev#5283](feast-dev#5283)) ([ecad170](feast-dev@ecad170)) * Offline store update pull_all_from_table_or_query to make timestampfield optional ([feast-dev#5281](feast-dev#5281)) ([4b94608](feast-dev@4b94608)) * Serialization version 2 deprecation notice ([feast-dev#5248](feast-dev#5248)) ([327d99d](feast-dev@327d99d)) * Vector length definition moved to Feature View from Config ([feast-dev#5289](feast-dev#5289)) ([d8f1c97](feast-dev@d8f1c97))

Manisha Sudhir added 2 commits August 16, 2023 09:28

Added milvus write batch functionality

cfde2f7

Fixed tests

a8e5c79

michaelbackes reviewed Aug 16, 2023

View reviewed changes

modifying _format_data_for_milvus based on PR comment

97c4789

michaelbackes approved these changes Aug 17, 2023

View reviewed changes

Manisha4 and others added 7 commits August 16, 2023 19:15

Merge branch 'master' into msudhir/add_vector_online_write_batch

f3d5af8

Update test_milvus_online_store.py

a6518ce

reverting yield commit

Revert "modifying _format_data_for_milvus based on PR comment"

1e375b4

This reverts commit 97c4789.

fixed test

423ca81

removing milvus online setup

64396e2

reverted changes back

880efed

removing the dynamic schema todo comment

cf9a21f

Manisha4 merged commit e915e8d into master Aug 17, 2023

piket pushed a commit that referenced this pull request Aug 26, 2025

fix: Java PR #17

49da810

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added milvus write batch functionality #17

Added milvus write batch functionality #17

Uh oh!

Manisha4 commented Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

Manisha4 Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

piket Aug 16, 2023 •

edited

Loading

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

Manisha4 Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023 •

edited

Loading

Uh oh!

Manisha4 Aug 16, 2023

Uh oh!

michaelbackes Aug 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Added milvus write batch functionality #17

Added milvus write batch functionality #17

Uh oh!

Conversation

Manisha4 commented Aug 16, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

piket Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelbackes Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

piket Aug 16, 2023 •

edited

Loading

michaelbackes Aug 16, 2023 •

edited

Loading