Skip to content

Fix Docker Compose tests for Feast 0.5 #725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woop merged 2 commits into from
May 21, 2020
Merged

Fix Docker Compose tests for Feast 0.5 #725

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d6faa77
Update Docker Compose tests to Feast 0.5
woop May 21, 2020
4dd7908
Set lower job polling rate for Docker Compose
woop May 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion infra/docker-compose/.env.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# General
COMPOSE_PROJECT_NAME=feast
FEAST_VERSION=latest
FEAST_REPOSITORY_VERSION=v0.4.7
FEAST_REPOSITORY_VERSION=v0.5-branch

# Feast Core
FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core
FEAST_CORE_CONFIG=direct-runner.yml
FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=placeholder.json
FEAST_STREAM_OPTIONS_BOOTSTRAPSERVERS=kafka:9092
FEAST_JOBS_POLLING_INTERVAL_MILLISECONDS=10

# Feast Serving Batch (BigQuery)
FEAST_BATCH_SERVING_CONFIG=batch-serving.yml
Expand Down
3 changes: 0 additions & 3 deletions infra/docker-compose/core/direct-runner.yml
View file
Open in desktop

This file was deleted.

2 changes: 2 additions & 0 deletions infra/docker-compose/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ services:
environment:
DB_HOST: db
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json
FEAST_STREAM_OPTIONS_BOOTSTRAPSERVERS: ${FEAST_STREAM_OPTIONS_BOOTSTRAPSERVERS}
FEAST_JOBS_POLLING_INTERVAL_MILLISECONDS: ${FEAST_JOBS_POLLING_INTERVAL_MILLISECONDS}
restart: on-failure
depends_on:
- db
Expand Down
3 changes: 3 additions & 0 deletions infra/docker-compose/jupyter/startup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ set -ex
# Clone Feast repository into Jupyter container
git clone -b ${FEAST_REPOSITORY_VERSION} --single-branch https://github.com/feast-dev/feast.git || true

# Install Python dependencies
make -C feast/ compile-protos-python

# Install CI requirements (only needed for running tests)
pip install -r feast/sdk/python/requirements-ci.txt

Expand Down
21 changes: 10 additions & 11 deletions infra/docker-compose/serving/online-serving.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
feast:
core-host: core
core-grpc-port: 6565
store:
config-path: /etc/feast/store.yml
jobs:
store-type: REDIS
store-options:
host: redis
port: 6379

grpc:
port: 6566
active_store: online
stores:
- name: online
type: REDIS
config:
host: redis
port: 6379
subscriptions:
- name: "*"
project: "*"
17 changes: 14 additions & 3 deletions infra/scripts/test-docker-compose.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,20 @@ Running Docker Compose tests with pytest at 'tests/e2e'
============================================================
"

clean_up () {
ARG=$?

# Shut down docker-compose images
docker-compose -f docker-compose.yml -f docker-compose.online.yml down

# Remove configuration file
rm .env

exit $ARG
}

trap clean_up EXIT

export PROJECT_ROOT_DIR=$(git rev-parse --show-toplevel)
export COMPOSE_INTERACTIVE_NO_CLI=1

Expand All @@ -30,6 +44,3 @@ ${PROJECT_ROOT_DIR}/infra/scripts/wait-for-it.sh ${JUPYTER_DOCKER_CONTAINER_IP_A

# Run e2e tests for Redis
docker exec feast_jupyter_1 bash -c 'cd feast/tests/e2e/ && pytest -s basic-ingest-redis-serving.py --core_url core:6565 --serving_url=online-serving:6566'

# Shut down docker-compose images
docker-compose -f docker-compose.yml -f docker-compose.online.yml down