Skip to content

fix: Add gRPC timeout and keepalive to RemoteRegistry - #6698

Open
Arthur-plg wants to merge 3 commits into
feast-dev:masterfrom
Arthur-plg:fix/remote-registry-grpc-timeout
Open

fix: Add gRPC timeout and keepalive to RemoteRegistry#6698
Arthur-plg wants to merge 3 commits into
feast-dev:masterfrom
Arthur-plg:fix/remote-registry-grpc-timeout

Conversation

@Arthur-plg

@Arthur-plg Arthur-plg commented Aug 2, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

Configures a default deadline (timeout) and transport keepalive options (keepalive_time_ms, keepalive_timeout_ms) for the client-side gRPC channel in RemoteRegistry.

Without these configurations, network drops or silent firewall cuts (blackholes) after a TCP connection is established cause all registry calls to hang indefinitely in epoll_wait without raising any errors or timeouts.

Which issue(s) this PR fixes:

Fixes #6665

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

NONE

apelong added 2 commits August 2, 2026 21:12
Signed-off-by: Pelong <arthur.pelong@ecl22.ec-lyon.fr>
Signed-off-by: Pelong <arthur.pelong@ecl22.ec-lyon.fr>
@Arthur-plg
Arthur-plg requested a review from a team as a code owner August 2, 2026 19:35

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new tuning fields are unconstrained Optional[int] values. timeout=0 or a negative keepalive interval/timeout is accepted by RemoteRegistryConfig and only fails later when gRPC constructs or uses the channel, turning a config typo into a runtime startup failure. We should validate positive values at config load (and add tests for zero/negative inputs); if None is intended to mean no deadline, the docs should also avoid calling this a default timeout.

Signed-off-by: Arthur Pelong <arthurpelong@gmail.com>
@Arthur-plg

Copy link
Copy Markdown
Author

Thanks for the feedback! I've added a Pydantic field_validator to RemoteRegistryConfig to ensure that timeout, keepalive_time_ms, and keepalive_timeout_ms are strictly positive at config load time.
I've also added unit tests validating that zero/negative inputs throw a ValidationError, and updated the documentation to clarify that calls will have no deadline if these values are not set.

Required when the connection address differs from the service hostname,
e.g. when connecting through a tunnel or proxy for local development. """

timeout: Optional[int] = None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be Optional[float] (or Optional[Union[int, float]]) ?

""" int: Period in milliseconds after which a keepalive ping is sent on the transport. Must be strictly positive. """

keepalive_timeout_ms: Optional[int] = None
""" int: Timeout in milliseconds for keepalive ping acknowledgement. Must be strictly positive. """

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be keepalive_permit_without_calls also ?

@codecov-commenter

codecov-commenter commented Aug 14, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.79%. Comparing base (a1e6fc2) to head (019af7e).
⚠️ Report is 15 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6698      +/-   ##
==========================================
+ Coverage   46.77%   46.79%   +0.02%     
==========================================
  Files         414      414              
  Lines       50191    50211      +20     
  Branches     7181     7185       +4     
==========================================
+ Hits        23475    23496      +21     
+ Misses      25077    25076       -1     
  Partials     1639     1639              
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.12% <100.00%> (+0.02%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/registry/remote.py 47.60% <100.00%> (+3.26%) ⬆️
...permissions/client/grpc_client_auth_interceptor.py 65.78% <100.00%> (+4.02%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4efb86c...019af7e. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote registry gRPC client sets no deadline and no keepalive, and neither is configurable: a blackholed connection hangs the caller indefinitely

5 participants