Skip to content

test: Retry CliRunner apply subprocess on timeout to reduce flakiness - #6592

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
nikolauspschuetz:test/stabilize-cli-runner-apply-retry
Jul 30, 2026
Merged

test: Retry CliRunner apply subprocess on timeout to reduce flakiness#6592
ntkathole merged 1 commit into
feast-dev:masterfrom
nikolauspschuetz:test/stabilize-cli-runner-apply-retry

Conversation

@nikolauspschuetz

Copy link
Copy Markdown
Contributor

What changed

CliRunner.run() (the helper that runs feast apply in a fresh child interpreter for unit tests) now retries once on subprocess.TimeoutExpired instead of treating the first timeout as terminal. The per-attempt timeout is unchanged (120s); the timeout message now includes the attempt counter.

Why

The unit-test workflow flaked on unit-test-python (3.11, macos-14) while setting up test_feature_server.py::test_push_and_get:

ERROR sdk/python/tests/unit/test_feature_server.py::test_push_and_get
  AssertionError: stdout: b''
  stderr: b"Command timed out after 120s: [\x27apply\x27]"
2259 passed, 47 skipped, 1 error

run() spawns a new Python interpreter (sys.executable feast.cli apply) — deliberately, for feature-repo import isolation — which has to import Feast (pandas/pyarrow/protobuf/...) from cold. On a loaded macOS runner that cold start can occasionally exceed the 120s guard. The same commit passed on macos-14 / 3.12 and on all three Ubuntu legs, which is the signature of an environmental cold-start stall rather than a code fault.

Crucially, commands routed through run() (e.g. apply) do not materialize, so they cannot trigger the Dask atexit hang that run_with_output() and the module docstring describe. A timeout here is therefore almost always transient slowness, not a deadlock — so a single retry with warm OS/page caches clears it, while a genuine repeated stall still fails once the attempts are exhausted. Retries are intentionally not added to run_with_output(), whose commands can legitimately hang and whose existing machinery recovers partial output rather than retrying.

This builds directly on the CliRunner subprocess stabilization in #6560.

Validation

  • uv run ruff check sdk/python/tests/utils/cli_repo_creator.py — passed
  • uv run ruff format --check sdk/python/tests/utils/cli_repo_creator.py — already formatted
  • Retry logic verified by monkeypatching subprocess.run:
    • transient timeout on attempt 1 → recovered on attempt 2 (returncode == 0)
    • always timing out → returncode == -1, message ... (attempt 2/2): [...], no exception raised
    • success on first attempt → exactly one subprocess call (no behavior change for the passing path)

Note: the full unit suite could not be collected on my machine (local venv lacks the google.cloud test extra pulled by conftest.py); the change is confined to the retry loop, which is covered by the targeted verification above.

Separate from #6591 (which surfaced this flake).

@nikolauspschuetz
nikolauspschuetz requested a review from a team as a code owner July 9, 2026 07:09
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from aadd2bb to e74df8e Compare July 14, 2026 06:04
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from e74df8e to 9aca1af Compare July 30, 2026 06:51
@codecov-commenter

codecov-commenter commented Jul 30, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.36%. Comparing base (0f149a9) to head (0860438).
⚠️ Report is 6 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6592      +/-   ##
==========================================
+ Coverage   46.35%   46.36%   +0.01%     
==========================================
  Files         414      414              
  Lines       50052    50072      +20     
  Branches     7151     7154       +3     
==========================================
+ Hits        23201    23218      +17     
- Misses      25229    25231       +2     
- Partials     1622     1623       +1     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 47.66% <ø> (+0.01%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a35fba...0860438. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

CliRunner.run() (used for 'feast apply' in unit tests) spawns a fresh child
interpreter that must import Feast (pandas/pyarrow/...) from cold. On loaded
CI runners — observed on unit-test-python (3.11, macos-14) — this can exceed
the 120s timeout, failing the test with:

    AssertionError: Command timed out after 120s: ['apply']

Unlike the materializing paths guarded by run_with_output(), commands routed
through run() do not materialize and so cannot trigger the Dask atexit hang the
timeout was added for. A timeout here is therefore almost always transient cold
-start slowness, not a genuine deadlock, so retrying once (with warm OS/page
caches) clears it while a real repeated stall still fails after the attempts
are exhausted.

Builds on the CliRunner subprocess stabilization in feast-dev#6560.

Signed-off-by: Nikolaus Schuetz <nikolauspschuetz@gmail.com>
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from 9aca1af to 0860438 Compare July 30, 2026 08:38
@ntkathole
ntkathole merged commit 6019bd6 into feast-dev:master Jul 30, 2026
18 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants