Skip to content

docs: Updated readme list of supported stores #6130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ntkathole wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: Updated readme list of supported stores #6130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,32 +182,53 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] [Couchbase (contrib plugin)](https://docs.feast.dev/reference/data-sources/couchbase)
* [x] [Athena (contrib plugin)](https://docs.feast.dev/reference/data-sources/athena)
* [x] [Clickhouse (contrib plugin)](https://docs.feast.dev/reference/data-sources/clickhouse)
* [x] [Oracle (contrib plugin)](https://docs.feast.dev/reference/data-sources/oracle)
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
* **Offline Stores**
* [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake)
* [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift)
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
* [x] [Azure Synapse + Azure SQL (contrib plugin)](https://docs.feast.dev/reference/offline-stores/mssql.md)
* [x] [DuckDB](https://docs.feast.dev/reference/offline-stores/duckdb)
* [x] [Dask](https://docs.feast.dev/reference/offline-stores/dask)
* [x] [Remote](https://docs.feast.dev/reference/offline-stores/remote-offline-store)
* [x] [Azure Synapse + Azure SQL (contrib plugin)](https://docs.feast.dev/reference/offline-stores/mssql)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
* [x] [Trino (contrib plugin)](https://docs.feast.dev/reference/offline-stores/trino)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [Couchbase (contrib plugin)](https://docs.feast.dev/reference/offline-stores/couchbase)
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
* [x] [Athena (contrib plugin)](https://docs.feast.dev/reference/offline-stores/athena)
* [x] [Clickhouse (contrib plugin)](https://docs.feast.dev/reference/offline-stores/clickhouse)
* [x] [Ray (contrib plugin)](https://docs.feast.dev/reference/offline-stores/ray)
* [x] [Oracle (contrib plugin)](https://docs.feast.dev/reference/offline-stores/oracle)
* [x] [Hybrid](https://docs.feast.dev/reference/offline-stores/hybrid)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-a-new-offline-store)
* **Online Stores**
* [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake)
* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)
* [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly)
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
* [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly)
* [x] [Remote](https://docs.feast.dev/reference/online-stores/remote)
* [x] [Postgres](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [HBase](https://docs.feast.dev/reference/online-stores/hbase)
* [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra)
* [x] [ScyllaDB](https://docs.feast.dev/reference/online-stores/scylladb)
* [x] [MySQL](https://docs.feast.dev/reference/online-stores/mysql)
* [x] [Hazelcast](https://docs.feast.dev/reference/online-stores/hazelcast)
* [x] [Elasticsearch](https://docs.feast.dev/reference/online-stores/elasticsearch)
* [x] [SingleStore](https://docs.feast.dev/reference/online-stores/singlestore)
* [x] [Couchbase](https://docs.feast.dev/reference/online-stores/couchbase)
* [x] [MongoDB](https://docs.feast.dev/reference/online-stores/mongodb)
* [x] [Qdrant (vector store)](https://docs.feast.dev/reference/online-stores/qdrant)
* [x] [Milvus (vector store)](https://docs.feast.dev/reference/online-stores/milvus)
* [x] [Faiss (vector store)](https://docs.feast.dev/reference/online-stores/faiss)
* [x] [Hybrid](https://docs.feast.dev/reference/online-stores/hybrid)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra)
* [x] [ScyllaDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/scylladb)
* [x] [Couchbase (contrib plugin)](https://docs.feast.dev/reference/online-stores/couchbase)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-support-for-a-new-online-store)
* **Feature Engineering**
* [x] On-demand Transformations (On Read) (Beta release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
Expand Down
9 changes: 9 additions & 0 deletions docs/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@
* [Azure Synapse + Azure SQL (contrib)](reference/data-sources/mssql.md)
* [Couchbase (contrib)](reference/data-sources/couchbase.md)
* [Oracle (contrib)](reference/data-sources/oracle.md)
* [Athena (contrib)](reference/data-sources/athena.md)
* [Clickhouse (contrib)](reference/data-sources/clickhouse.md)
* [Offline stores](reference/offline-stores/README.md)
* [Overview](reference/offline-stores/overview.md)
* [Dask](reference/offline-stores/dask.md)
Expand All @@ -114,7 +116,9 @@
* [Clickhouse (contrib)](reference/offline-stores/clickhouse.md)
* [Ray (contrib)](reference/offline-stores/ray.md)
* [Oracle (contrib)](reference/offline-stores/oracle.md)
* [Athena (contrib)](reference/offline-stores/athena.md)
* [Remote Offline](reference/offline-stores/remote-offline-store.md)
* [Hybrid](reference/offline-stores/hybrid.md)
* [Online stores](reference/online-stores/README.md)
* [Overview](reference/online-stores/overview.md)
* [SQLite](reference/online-stores/sqlite.md)
Expand All @@ -126,6 +130,7 @@
* [Bigtable](reference/online-stores/bigtable.md)
* [Remote](reference/online-stores/remote.md)
* [PostgreSQL](reference/online-stores/postgres.md)
* [HBase](reference/online-stores/hbase.md)
* [Cassandra + Astra DB](reference/online-stores/cassandra.md)
* [Couchbase](reference/online-stores/couchbase.md)
* [MySQL](reference/online-stores/mysql.md)
Expand All @@ -134,6 +139,10 @@
* [SingleStore](reference/online-stores/singlestore.md)
* [Milvus](reference/online-stores/milvus.md)
* [MongoDB](reference/online-stores/mongodb.md)
* [Elasticsearch](reference/online-stores/elasticsearch.md)
* [Qdrant](reference/online-stores/qdrant.md)
* [Faiss](reference/online-stores/faiss.md)
* [Hybrid](reference/online-stores/hybrid.md)
* [Registries](reference/registries/README.md)
* [Local](reference/registries/local.md)
* [S3](reference/registries/s3.md)
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/data-sources/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ Please see [Data Source](../../getting-started/concepts/data-ingestion.md) for a
[clickhouse.md](clickhouse.md)
{% endcontent-ref %}

{% content-ref url="athena.md" %}
[athena.md](athena.md)
{% endcontent-ref %}

{% content-ref url="oracle.md" %}
[oracle.md](oracle.md)
{% endcontent-ref %}
37 changes: 37 additions & 0 deletions docs/reference/data-sources/athena.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Athena source (contrib)

## Description

Athena data sources are AWS Athena tables or views.
These can be specified either by a table reference or a SQL query.

## Disclaimer

The Athena data source does not achieve full test coverage.
Please do not assume complete stability.

## Examples

Defining an Athena source:

```python
from feast.infra.offline_stores.contrib.athena_offline_store.athena_source import (
AthenaSource,
)

driver_stats_source = AthenaSource(
name="driver_hourly_stats",
table="driver_hourly_stats",
database="my_database",
data_source="AwsDataCatalog",
timestamp_field="event_timestamp",
created_timestamp_column="created",
)
```

The full set of configuration options is available [here](https://rtd.feast.dev/en/master/#feast.infra.offline_stores.contrib.athena_offline_store.athena_source.AthenaSource).

## Supported Types

Athena data sources support standard Athena types mapped through the AWS Athena API.
For a comparison against other batch data sources, please see [here](overview.md#functionality-matrix).
16 changes: 16 additions & 0 deletions docs/reference/offline-stores/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,19 @@ Please see [Offline Store](../../getting-started/components/offline-store.md) fo
{% content-ref url="oracle.md" %}
[oracle.md](oracle.md)
{% endcontent-ref %}

{% content-ref url="athena.md" %}
[athena.md](athena.md)
{% endcontent-ref %}

{% content-ref url="clickhouse.md" %}
[clickhouse.md](clickhouse.md)
{% endcontent-ref %}

{% content-ref url="remote-offline-store.md" %}
[remote-offline-store.md](remote-offline-store.md)
{% endcontent-ref %}

{% content-ref url="hybrid.md" %}
[hybrid.md](hybrid.md)
{% endcontent-ref %}
66 changes: 66 additions & 0 deletions docs/reference/offline-stores/athena.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Athena offline store (contrib)

## Description

The Athena offline store provides support for reading [AthenaSources](../data-sources/athena.md).
* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe.

## Disclaimer

The Athena offline store does not achieve full test coverage.
Please do not assume complete stability.

## Getting started
In order to use this offline store, you'll need to run `pip install 'feast[aws]'`.

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_project
registry: data/registry.db
provider: local
offline_store:
type: athena
data_source: AwsDataCatalog
region: us-east-1
database: my_database
workgroup: primary
online_store:
path: data/online_store.db
```
{% endcode %}

The full set of configuration options is available in [AthenaOfflineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.offline_stores.contrib.athena_offline_store.athena.AthenaOfflineStoreConfig).

## Functionality Matrix

The set of functionality supported by offline stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Athena offline store.

| | Athena |
| :----------------------------------------------------------------- |:-------|
| `get_historical_features` (point-in-time correct join) | yes |
| `pull_latest_from_table_or_query` (retrieve latest feature values) | yes |
| `pull_all_from_table_or_query` (retrieve a saved dataset) | yes |
| `offline_write_batch` (persist dataframes to offline store) | no |
| `write_logged_features` (persist logged features to offline store) | yes |

Below is a matrix indicating which functionality is supported by `AthenaRetrievalJob`.

| | Athena |
| ----------------------------------------------------- |--------|
| export to dataframe | yes |
| export to arrow table | yes |
| export to arrow batches | no |
| export to SQL | yes |
| export to data lake (S3, GCS, etc.) | no |
| export to data warehouse | no |
| export as Spark dataframe | no |
| local execution of Python-based on-demand transforms | yes |
| remote execution of Python-based on-demand transforms | no |
| persist results in the offline store | yes |
| preview the query plan before execution | yes |
| read partitioned data | yes |

To compare this set of functionality against other offline stores, please see the full [functionality matrix](overview.md#functionality-matrix).
24 changes: 24 additions & 0 deletions docs/reference/online-stores/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
[postgres.md](postgres.md)
{% endcontent-ref %}

{% content-ref url="hbase.md" %}
[hbase.md](hbase.md)
{% endcontent-ref %}

{% content-ref url="cassandra.md" %}
[cassandra.md](cassandra.md)
{% endcontent-ref %}
Expand Down Expand Up @@ -69,3 +73,23 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
{% content-ref url="singlestore.md" %}
[singlestore.md](singlestore.md)
{% endcontent-ref %}

{% content-ref url="elasticsearch.md" %}
[elasticsearch.md](elasticsearch.md)
{% endcontent-ref %}

{% content-ref url="qdrant.md" %}
[qdrant.md](qdrant.md)
{% endcontent-ref %}

{% content-ref url="milvus.md" %}
[milvus.md](milvus.md)
{% endcontent-ref %}

{% content-ref url="faiss.md" %}
[faiss.md](faiss.md)
{% endcontent-ref %}

{% content-ref url="hybrid.md" %}
[hybrid.md](hybrid.md)
{% endcontent-ref %}
57 changes: 57 additions & 0 deletions docs/reference/online-stores/faiss.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Faiss online store

## Description

The [Faiss](https://github.com/facebookresearch/faiss) online store provides support for materializing feature values and performing vector similarity search using Facebook AI Similarity Search (Faiss). Faiss is a library for efficient similarity search and clustering of dense vectors, making it well-suited for use cases involving embeddings and nearest-neighbor lookups.

## Getting started
In order to use this online store, you'll need to install the Faiss dependency. E.g.

`pip install 'feast[faiss]'`

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: feast.infra.online_stores.faiss_online_store.FaissOnlineStore
dimension: 128
index_path: data/faiss_index
index_type: IVFFlat # optional, default: IVFFlat
nlist: 100 # optional, default: 100
```
{% endcode %}

**Note:** Faiss is not registered as a named online store type. You must use the fully qualified class path as the `type` value.

The full set of configuration options is available in [FaissOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.faiss_online_store.FaissOnlineStoreConfig).

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Faiss online store.

| | Faiss |
|:----------------------------------------------------------|:------|
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |
| vector similarity search | yes |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
56 changes: 56 additions & 0 deletions docs/reference/online-stores/hbase.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# HBase online store

## Description

The [HBase](https://hbase.apache.org/) online store provides support for materializing feature values into an Apache HBase database for serving online features in real-time.

* Each feature view is mapped to an HBase table
* Connects to HBase via the Thrift server using [happybase](https://happybase.readthedocs.io/)

## Getting started
In order to use this online store, you'll need to run `pip install 'feast[hbase]'`.

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: hbase
host: localhost
port: "9090"
connection_pool_size: 4 # optional
protocol: binary # optional
transport: buffered # optional
```
{% endcode %}

The full set of configuration options is available in [HbaseOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.hbase_online_store.hbase.HbaseOnlineStoreConfig).

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the HBase online store.

| | HBase |
| :-------------------------------------------------------- | :---- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
Loading
Loading