Skip to content

Change RedisBackedJobService to use a connection pool#439

Merged
feast-ci-bot merged 6 commits intofeast-dev:masterfrom
zhilingc:redis-conn-pool
Jan 21, 2020
Merged

Change RedisBackedJobService to use a connection pool#439
feast-ci-bot merged 6 commits intofeast-dev:masterfrom
zhilingc:redis-conn-pool

Conversation

@zhilingc
Copy link
Copy Markdown
Collaborator

@zhilingc zhilingc commented Jan 21, 2020

What this PR does / why we need it:
Current implementation of connection to jobs redis is using a client that upon failure does not refresh its connection to the db - so if anything goes awry batch retrieval is unusable until the instance is restarted to manually establish a new connection with redis.

This PR changes RedisBackedJobService to use a connection pool instead.

Which issue(s) this PR fixes:

N/A

Does this PR introduce a user-facing change?:

NONE

@zhilingc zhilingc requested a review from pradithya as a code owner January 21, 2020 03:13
@zhilingc zhilingc changed the title Change job service connection to use a connection pool Change RedisBackedJobService to use a connection pool Jan 21, 2020
@zhilingc zhilingc requested a review from khorshuheng January 21, 2020 03:13
@khorshuheng
Copy link
Copy Markdown
Collaborator

khorshuheng commented Jan 21, 2020

/lgtm

@khorshuheng
Copy link
Copy Markdown
Collaborator

khorshuheng commented Jan 21, 2020

/approve

@feast-ci-bot feast-ci-bot merged commit 0a8987e into feast-dev:master Jan 21, 2020
@feast-ci-bot
Copy link
Copy Markdown
Collaborator

feast-ci-bot commented Jan 21, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khorshuheng, zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@brandon733
Copy link
Copy Markdown

brandon733 commented Jan 24, 2020 •
edited
Loading

not sure if this is bc of this change, but to get minikube working w basic.ipynb, I had to do:

diff --git a/infra/charts/feast/values-demo.yaml b/infra/charts/feast/values-demo.yaml
index fad4bc0..eee063d 100644
--- a/infra/charts/feast/values-demo.yaml
+++ b/infra/charts/feast/values-demo.yaml
@@ -62,6 +62,9 @@ feast-serving-online:
   store.yaml:
     name: redis
     type: REDIS
+    redis_config:
+      host: feast.example.com
+      port: 32101
     subscriptions:
       - name: "*"
         project: "*"

o/w ingest() would fail w/ cannot conn to redis

@khorshuheng
Copy link
Copy Markdown
Collaborator

khorshuheng commented Jan 25, 2020

@b133t Which image version are you using?

@brandon733
Copy link
Copy Markdown

brandon733 commented Jan 27, 2020 •
edited
Loading

0.4.3
minikube setup as detailed in https://docs.feast.dev/getting-started/installing-feast#minikube
then running basic.ipynb in source repo w/ jupyter (without batch)
fails on "7. Ingest data into Feast for a feature set", unless make change above (and s/feast.example.com/$(minikube ip)/ + rerun helm)

khorshuheng pushed a commit that referenced this pull request Feb 13, 2020
Change RedisBackedJobService to use a connection pool (#439)
dcaccaa 
* Change job service connection to use a connection pool

* Close connection

* Remove isConnected condition

* Add simple test to test that pool recovers broken connections

* Catch JedisExceptions separately

* Catch only JedisConnectionException
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidheryanto davidheryanto davidheryanto left review comments

@pradithya pradithya Awaiting requested review from pradithya

@khorshuheng khorshuheng Awaiting requested review from khorshuheng

Assignees

@khorshuheng khorshuheng

Labels

approved lgtm size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zhilingc @khorshuheng @feast-ci-bot @brandon733 @davidheryanto