Skip to content

feat: Postgres supported as Registry, Online store, and Offline store #2401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 24 commits into from
Apr 20, 2022
Merged

feat: Postgres supported as Registry, Online store, and Offline store #2401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ed18f7e
Feast-postgres added to Feast repo
nossrannug Mar 13, 2022
bc8a630
Add a template for postgres to allow feast init -t postgres
nossrannug Mar 20, 2022
1a705e3
Split contrib repo configuration to separate file for each
nossrannug Mar 20, 2022
643f911
Moved the postgres type maps into feast.type_map
nossrannug Mar 22, 2022
15e71e7
Upgrade to latest feast, get postgres tests running, pip lock files
nossrannug Apr 19, 2022
953c3e7
Fix setup.py
kevjumba Apr 20, 2022
66c4699
Fix the ci requirements
kevjumba Apr 20, 2022
d0b5081
Fix repo_config for rest of sources
kevjumba Apr 20, 2022
c4de36e
Make backwards compatible with python 3.8-
kevjumba Apr 20, 2022
979e326
Fix
kevjumba Apr 20, 2022
199850e
Fix
kevjumba Apr 20, 2022
885d789
address comments
adchia Apr 20, 2022
ad947a9
update lock files
adchia Apr 20, 2022
1f246ad
fix
adchia Apr 20, 2022
f625910
lock files
adchia Apr 20, 2022
c6d93cd
fix
adchia Apr 20, 2022
e6c39ff
cleanup
adchia Apr 20, 2022
caad6c0
missed refactor name
adchia Apr 20, 2022
a0f9476
Remove snowflake references from test.py for postgres
nossrannug Apr 20, 2022
7a792a5
update repo to match new API
adchia Apr 20, 2022
8e1509d
Add documentation
adchia Apr 20, 2022
8e6ec1e
rename utils
adchia Apr 20, 2022
3522e23
update roadmap docs
adchia Apr 20, 2022
72e9e3e
update documentation
adchia Apr 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,19 @@ test-python-integration-container:
test-python-universal-contrib:
PYTHONPATH='.' FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.contrib_repo_configuration FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests

test-python-universal-postgres:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest --integration --universal \
-k "not test_historical_retrieval_fails_on_validation and \
not test_historical_retrieval_with_validation and \
not test_historical_features_persisting and \
not test_historical_retrieval_fails_on_validation and \
not test_universal_cli" \
sdk/python/tests

test-python-universal-local:
FEAST_USAGE=False IS_TEST=True FEAST_IS_LOCAL_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests

Expand Down
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
* [ ] HTTP source
* **Offline Stores**
Expand All @@ -156,9 +156,9 @@ The list below contains the functionality that contributors are planning to deve
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
* **Online Stores**
Expand All @@ -167,7 +167,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [ ] Bigtable (in progress)
* [ ] Cassandra
Expand Down
12 changes: 6 additions & 6 deletions docs/getting-started/third-party-integrations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ Don't see your offline store or online store of choice here? Check out our guide
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
* [ ] HTTP source

Expand All @@ -31,9 +31,9 @@ Don't see your offline store or online store of choice here? Check out our guide
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)

Expand All @@ -44,7 +44,7 @@ Don't see your offline store or online store of choice here? Check out our guide
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [ ] Bigtable (in progress)
* [ ] Cassandra
Expand Down
25 changes: 25 additions & 0 deletions docs/reference/data-sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# PostgreSQL

## Description

**NOTE**: The Postgres plugin is a contrib plugin. This means it may not be fully stable.


The PostgreSQL data source allows for the retrieval of historical feature values from a PostgreSQL database for building training datasets as well as materializing features into an online store.

## Examples

Defining a Postgres source

```python
from feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source import (
PostgreSQLSource,
)

driver_stats_source = PostgreSQLSource(
name="feast_driver_hourly_stats",
query="SELECT * FROM feast_driver_hourly_stats",
timestamp_field="event_timestamp",
created_timestamp_column="created",
)
```
2 changes: 1 addition & 1 deletion docs/reference/data-sources/spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Spark
# Spark (contrib)

## Description

Expand Down
34 changes: 34 additions & 0 deletions docs/reference/offline-stores/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# PostgreSQL (contrib)

## Description

The PostgreSQL offline store is an offline store that provides support for reading [PostgreSQL](../data-sources/postgres.md) data sources.


**DISCLAIMER**: This PostgreSQL offline store still does not achieve full test coverage.

* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be converted to a Spark dataframe and processed as a temporary view.
* A `SparkRetrievalJob` is returned when calling `get_historical_features()`.
* This allows you to call
* `to_df` to retrieve the pandas dataframe.
* `to_arrow` to retrieve the dataframe as a PyArrow table.

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_project
registry: data/registry.db
provider: local
offline_store:
type: postgres
host: DB_HOST
port: DB_PORT
database: DB_NAME
db_schema: DB_SCHEMA
user: DB_USERNAME
password: DB_PASSWORD
online_store:
path: data/online_store.db
```
{% endcode %}
2 changes: 1 addition & 1 deletion docs/reference/offline-stores/spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Spark
# Spark (contrib)

## Description

Expand Down
27 changes: 27 additions & 0 deletions docs/reference/online-stores/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# PostgreSQL (contrib)

## Description

The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.

* Only the latest feature values are persisted

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: postgres
host: DB_HOST
port: DB_PORT
database: DB_NAME
db_schema: DB_SCHEMA
user: DB_USERNAME
password: DB_PASSWORD
```
{% endcode %}

Configuration options are available [here](https://rtd.feast.dev/en/latest/#feast.repo_config.SqliteOnlineStoreConfig).
12 changes: 6 additions & 6 deletions docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
* [ ] HTTP source
* **Offline Stores**
Expand All @@ -24,9 +24,9 @@ The list below contains the functionality that contributors are planning to deve
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
* **Online Stores**
Expand All @@ -35,7 +35,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [ ] Bigtable (in progress)
* [ ] Cassandra
Expand Down
21 changes: 20 additions & 1 deletion sdk/python/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ Trino Source
:members:
:exclude-members: TrinoOptions

PostgreSQL Source
------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source
:members:
:exclude-members: PostgreSQLOptions

File Source
------------------
Expand Down Expand Up @@ -192,6 +198,12 @@ Trino Offline Store
.. automodule:: feast.infra.offline_stores.contrib.trino_offline_store.trino
:members:

PostgreSQL Offline Store
------------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres
:members:


Online Store
==================
Expand Down Expand Up @@ -223,4 +235,11 @@ Redis Online Store

.. automodule:: feast.infra.online_stores.redis
:members:
:noindex:
:noindex:

PostgreSQL Online Store
-----------------------

.. automodule:: feast.infra.online_stores.contrib.postgres
:members:
:noindex:
29 changes: 29 additions & 0 deletions ...ython/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
feast.infra.offline\_stores.contrib.postgres\_offline\_store package
====================================================================

Submodules
----------

feast.infra.offline\_stores.contrib.postgres\_offline\_store.postgres module
----------------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres
:members:
:undoc-members:
:show-inheritance:

feast.infra.offline\_stores.contrib.postgres\_offline\_store.postgres\_source module
------------------------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store
:members:
:undoc-members:
:show-inheritance:
9 changes: 9 additions & 0 deletions sdk/python/docs/source/feast.infra.offline_stores.contrib.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Subpackages
.. toctree::
:maxdepth: 4

feast.infra.offline_stores.contrib.postgres_offline_store
feast.infra.offline_stores.contrib.spark_offline_store
feast.infra.offline_stores.contrib.trino_offline_store

Expand All @@ -21,6 +22,14 @@ feast.infra.offline\_stores.contrib.contrib\_repo\_configuration module
:undoc-members:
:show-inheritance:

feast.infra.offline\_stores.contrib.postgres\_repo\_configuration module
------------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_repo_configuration
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

Expand Down
21 changes: 21 additions & 0 deletions sdk/python/docs/source/feast.infra.online_stores.contrib.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
feast.infra.online\_stores.contrib package
==========================================

Submodules
----------

feast.infra.online\_stores.contrib.postgres module
--------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.postgres
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.online_stores.contrib
:members:
:undoc-members:
:show-inheritance:
8 changes: 8 additions & 0 deletions sdk/python/docs/source/feast.infra.online_stores.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
feast.infra.online\_stores package
==================================

Subpackages
-----------

.. toctree::
:maxdepth: 4

feast.infra.online_stores.contrib

Submodules
----------

Expand Down
29 changes: 29 additions & 0 deletions sdk/python/docs/source/feast.infra.utils.postgres.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
feast.infra.utils.postgres package
==================================

Submodules
----------

feast.infra.utils.postgres.connection\_utils module
---------------------------------------------------

.. automodule:: feast.infra.utils.postgres.connection_utils
:members:
:undoc-members:
:show-inheritance:

feast.infra.utils.postgres.postgres\_config module
--------------------------------------------------

.. automodule:: feast.infra.utils.postgres.postgres_config
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.utils.postgres
:members:
:undoc-members:
:show-inheritance:
8 changes: 8 additions & 0 deletions sdk/python/docs/source/feast.infra.utils.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
feast.infra.utils package
=========================

Subpackages
-----------

.. toctree::
:maxdepth: 4

feast.infra.utils.postgres

Submodules
----------

Expand Down
Loading