[Bug]: Random timeout failures in CI

Bug summary

We have been struggling with flaky timeout behavior for some time. See #29797. At some point we thought, we had fixed it though updating the runner, but it sometimes happens again. So it's not eventually solved. Possibly we have to revive #29981 and/or dig deeper.

Failures look like

>               raise TimeoutExpired(self.args, orig_timeout)
E               subprocess.TimeoutExpired: Command '['C:\\hostedtoolcache\\windows\\Python\\3.12.10\\x64\\python.exe', '-c', "import importlib.util;_spec = importlib.util.spec_from_file_location('matplotlib.tests.test_backends_interactive', 'D:\\\\a\\\\1\\\\s\\\\lib\\\\matplotlib\\\\tests\\\\test_backends_interactive.py');_module = importlib.util.module_from_spec(_spec);_spec.loader.exec_module(_module);_module._impl_test_lazy_auto_backend_selection()"]' timed out after 20 seconds

Code for reproduction

For a start, let's track in which pipelines and for which tests the timeout can be observed:

Pytest Windows_py311 - test_lazy_auto_backend_selection ref
Pytest Windows_py313 - test_interactive_backend[toolbar2-MPLBACKEND=tkagg-BACKEND_DEPS=tkinter] ref
Pytest Windows_py313 - test_fontcache_thread_safe ref
Pytest Windows_py313 - test_fallback_to_different_backend ref
Pytest Windows_py313 - test_blitting_events[MPLBACKEND=tkagg-BACKEND_DEPS=tkinter] ref
Pytest Windows_py313 test_blit ref
Pytest Windows_py313 test_fallback_to_different_backend ref
Pytest Windows_py313 test_backend_getattr[matplotlib.backends.backend_agg] ref
Pytest Windows_py312 test_interactive_thread_safety[MPLBACKEND=tkagg-BACKEND_DEPS=tkinter] ref
Pytest Windows_py313 test_interactive_timers[MPLBACKEND=tkagg-BACKEND_DEPS=tkinter] ref
Pytest Windows_py312 test_fontcache_thread_safe ref
Pytest Windows_py313 test_blit ref
Pytest Windows_py312 test_qt_missing ref

Observations

seems only to occur on windows, but with all python versions (3.11-3.13)
if it fails it's typically one test, but up to 4 timed out tests have been observed in a single pytest run
the Majority of PRs have at least one timeout
it happens in different tests every time but the common part is that they create subprocesses

@timhoffm

@timhoffm I'm not sure if you look at Gitter these days but Tom posted something there yesterday about the timeout/subprocess problem.

@rcomer

@rcomer thanks for the hint. I actually don't regularly check Gitter since it doesn't work with mobile browsers anymore.

Thanks for tracking this—I've reproduced similar subprocess timeouts locally on Windows (Python 3.12 via micromamba) in test_backends_interactive and test_fontcache_thread_safe. Tests hang on tkagg backend init despite Agg fallback.

Local observations:

Pipes from pytest subprocess calls block when CREATE_NO_WINDOW is missing.
Frequency: ~30% of runs in my CI-like setup (parallel pytest -n auto).

Quick fixes to test:

Add creationflags=subprocess.CREATE_NO_WINDOW to affected test subprocess calls.
Bump timeout to 40s for Windows: pytest --timeout=40 in azure-pipelines.yml.
Skip interactive backends in CI: MPLTESTARGS="-m 'not interactive'".

Happy to prototype a PR reviving #29981 with pipe fixes if this aligns—thoughts?

Thanks for looking into this. What I'm not clear: The Quick fixes to test and #29981 are four different solution approaches. What are you planing to do in the PR? Do you intend to evaluate the alternatives beforehand?

@timhoffm

Thanks for looking into this. What I'm not clear: The Quick fixes to test and #29981 are four different solution approaches. What are you planing to do in the PR? Do you intend to evaluate the alternatives beforehand?

Thanks @timhoffm—yeah, i am planning for one focused PR targeting subprocess flags only,

Add creationflags=subprocess.CREATE_NO_WINDOW to the pytest subprocess calls in lib/matplotlib/testing/decorators.py

This directly fixes the pipe blocking I see locally in the hanging tests

I'll benchmark it this weekend (100+ runs on my Windows setup with parallel pytest) to confirm it drops the ~30% failure rate to near-zero, then open a draft PR reviving the #29981 approach.

Config changes (timeout bumps, skipping tests) are just bandaids I'd only suggest if the pipe fix doesn't land.

Does focusing on the subprocess flags first make sense to you?

@timhoffm

Hi @timhoffm, just checking in on this PR #30866.
Is there anything blocking the review on my end (e.g., merge conflicts or missing tests)? Happy to fix anything to make your review easier
I am new contributor please guide me

@sanchit122006 please be patient. The majority of reviewers do this in their spare time. Response time may vary on priority, complexity of the topic and available time. Please also note that part of the world has Christmas holidays.

@timhoffm

@timhoffm thanks for guidance and sorry of inconvinience

@timhoffm

Hi @timhoffm,

Just following up on this #30886, Since the tests passed successfully and i analyse that azure pipelines tests are successful within 6 seconds on an average

Are we good to proceed with merging this to see if it resolves the flakiness on the main branch?

I'm happy to work if anything needed.

Any change needs two reviews from core developers. We are waiting for the second review.

@timhoffm

Understood, thanks for the clarification @timhoffm. I'll wait for the second review
let me know if the second reviewer needs anything.

And we have the counter-example: I rebased #30871 on main after this #30886 was merged, and the timeouts still occur:
https://dev.azure.com/matplotlib/matplotlib/_build/results?buildId=46442&view=logs&s=7cadbae7-7533-548a-becd-a2da10fc0146&j=f92c32cd-721a-536e-3a87-a0c010db3f7f

So #30886 is not a fix.

@timhoffm

i am really want to solve the issue of remaining timouts i really working on it
Are you want to proceed my surgical approach,
Give me the command chief @timhoffm

It may be worth considering image tests for formats other than png, because we use subprocess to convert them. But AFAIK we’ve never had a timeout in one.

@rcomer

I don't really care about funny, I care about efficient problem solving.

I agree that a surgical approach is sutable. However, the assumptions and details are incorrect. - This is a typical pattern in AI, and unfortunately we get more low quality contributions now that people can easily generate "a solution" via AI. This is why I react somewhat allergic to logically incorrect approaches. The value that contributors can bring is that they can understand and really think through a problem. Whether using AI in the background or not, #30851 (comment) was lacking in that.

To move forward, you need to identify the correct set of tests. My hypothesis is that only tests that spawn python with matplotlib are prone to hang. Put them all in one group (as @rcomer explained, we use two workers in CI, so splitting in multiple groups cannot reduce runtime). Then compre test run timing with and without that group. If the delay is acceptably, we roll it out and see whether it fixes the issue (there is no easy way in figuring out whether the change helps through local testing).

@timhoffm

Thankyou for explaining that much chief @timhoffm
i will work on with an without testing and report you

Status:

BUG: Fix Windows subprocess timeouts with CREATE_NO_WINDOW flag #30886 did not solve this. I have no clear data, but subjectively, timeouts have not substantially decreased after merge.
TST: always force the SETUPTOOLS_SCM version in test subprocesses #30947 substantially decreased the number of timeouts, but did not solve the issue either. - Likely it just reduced the number of timeouts by reducing the number of subprocesses we use.

Resolving the issue remains highly desirable.

I wonder if it's worth turning on verbose mode (pytest -v) in CI. Then we'd see what order the tests completed in and, in particular, what the other worker(s) were testing while a given test was timing out. This may or may not produce a clue to the problem.

Idea: there is some resource-intensive test that slows down anything that runs concurrently with it. We see the problem only in the subprocess-calling tests because those are the ones that have a time limit, but the subprocesses themselves are not the problem.

The slowest thing in this run is the setup of a test that does not use subprocess. I think there is no reason this setup should be any longer than for any other test that uses check_figures_equal and pandas.

@rcomer

@rcomer That analysis matches the behavior I'm seeing perfectly.

My "Strict Serialization" experiment failed because xdist_group only enforces serial execution within the group (on a single worker). It does not pause the other worker. So, while my PR successfully isolated the subprocess tests on Worker A, the "Bully" test you identified (the heavy pandas setup) was likely running simultaneously on Worker B, starving the system of resources.

Could you point me to the specific test file or name that you saw taking the longest in the logs?

To be clear, I'm not suggesting the pandas setup is heavy. I'm saying there is no reason that setup should have taken over a minute - there are other tests that use the exact same fixtures. So my idea is that something else happened at the same time to slow it down. But I have no idea what the something else is.

Can we detect whether a timed out test was hanging at a specific call for a longer time, or whether it was just slowly executing (or whether CPU utilization was high at that time)?

It looks like the flakes on Windows are happening because two heavy tasks are trying to run at the exact same time: the heavy Pandas setup in test_axes.py and the interactive subprocess tests. They end up fighting for resources, which causes the timeouts.

To fix this, I added --dist loadgroup and put both files in the same xdist group. This forces them to wait their turn and run on the same worker instead of conflicting with each other. It’s a simple scheduling change that stops the starvation issue without slowing down the rest of the test suite. see #31077

@timhoffm

@timhoffm I tested the grouped setup locally

Before ( with grouping occuring )

(mpl_stable) C:\Users\My\Documents\GitHub\matplotlib>python -m pytest -n 2 -vv --setup-show --durations=20 --durations-min=1.0 -k "pandas or check_figures_equal"
=========================================================================================================== test session starts ============================================================================================================
platform win32 -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0 -- C:\Users\My\.conda\envs\mpl_stable\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\My\Documents\GitHub\matplotlib
configfile: pyproject.toml
testpaths: lib
plugins: xdist-3.8.0
2 workers [20 items]   skipped
scheduling tests via LoadScheduling
[gw1] PASSED lib/matplotlib/tests/test_axes.py::test_pandas_indexing_dates
[gw0] XFAIL lib/matplotlib/tests/test_testing.py::test_check_figures_equal_extra_fig[png] <- lib\matplotlib\testing\decorators.py
lib/matplotlib/tests/test_testing.py::test_check_figures_equal_closed_fig[png] <- lib\matplotlib\testing\decorators.py

After diagnosis ( grouping not occuring )

(mpl_stable) C:\Users\My\Documents\GitHub\matplotlib>python -m pytest -n 2 --dist loadgroup -vv -k "test_bar_pandas_indexed or test_backend_interactive"
=========================================================================================================== test session starts ============================================================================================================
platform win32 -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0 -- C:\Users\My\.conda\envs\mpl_stable\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\My\Documents\GitHub\matplotlib
configfile: pyproject.toml
testpaths: lib
plugins: xdist-3.8.0
2 workers [1 item]     skipped
scheduling tests via LoadGroupScheduling

lib/matplotlib/tests/test_axes.py::test_bar_pandas_indexed@heavy_resource_group
[gw0] [100%] PASSED lib/matplotlib/tests/test_axes.py::test_bar_pandas_indexed@heavy_resource_group

and everything went smoothly—no timeouts and the grouping worked exactly as planned. It’s a good sign, but since it's only one run, I'm not ready to call it a total success just yet. I’m going to keep a close eye on the Windows CI runs and might flip on the debug logging if things get weird, just to be sure we've actually nailed the root cause.

if you can i go with permanent fix but this is good sign for now

Status update: Other kinds of flaky tests have been fixed meanwhile, but the subprocess timeouts remain - even tough it feels there are less often now.

Some recent examples:

Pytest Windows_py313 - test_canvas_focus ref
Pytest Windows_py312 - test_webagg_fallback[nbagg] ref
Pytest Windows_py312 - test_embedding ref
Pytest Windows_py312 - test_fontcache_thread_safe ref
Pytest Windows_py313 - test_backend_getattr[matplotlib.backends.backend_qt5agg] ref

My impression is that they are fewer since #31503, but I do not have hard data to back that up.

I think it was doing quite well, but lately it seems like we do have more Azure failures.

Navigation Menu

Repository navigation

[Bug]: Random timeout failures in CI #30851

Bug summary

Code for reproduction

Observations

Are we good to proceed with merging this to see if it resolves the flakiness on the main branch?

11 remaining items

Before ( with grouping occuring )

After diagnosis ( grouping not occuring )

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Random timeout failures in CI #30851

Description

Bug summary

Code for reproduction

Observations

Activity

rcomer commented on Dec 13, 2025

timhoffm commented on Dec 13, 2025

sanrishi commented on Dec 17, 2025

timhoffm commented on Dec 17, 2025

sanrishi commented on Dec 17, 2025

sanrishi commented on Dec 25, 2025

timhoffm commented on Dec 26, 2025

sanrishi commented on Dec 26, 2025

sanrishi commented on Jan 5, 2026

Are we good to proceed with merging this to see if it resolves the flakiness on the main branch?

timhoffm commented on Jan 6, 2026

sanrishi commented on Jan 6, 2026

timhoffm commented on Jan 9, 2026

11 remaining items

sanrishi commented on Jan 24, 2026

rcomer commented on Jan 24, 2026

timhoffm commented on Jan 24, 2026

sanrishi commented on Jan 24, 2026

timhoffm commented on Feb 2, 2026

rcomer commented on Feb 4, 2026

rcomer commented on Feb 4, 2026

sanrishi commented on Feb 4, 2026

rcomer commented on Feb 4, 2026

timhoffm commented on Feb 4, 2026

sanrishi commented on Feb 4, 2026

sanrishi commented on Feb 4, 2026

Before ( with grouping occuring )

After diagnosis ( grouping not occuring )

timhoffm commented on Apr 23, 2026

rcomer commented on Apr 23, 2026

QuLogic commented on Jun 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions