Repository navigation
Functional tests that create/update/delete resources are flaky #1033
Description
I realized the weird unrelated failure in #1020 (see https://travis-ci.org/python-gitlab/python-gitlab/jobs/652064210) was a flaky test for rate limits. I'm getting it locally sometimes but only rarely.
I haven't had the time to investigate but I'll try to have a look at some point. Maybe something in the environment slows down the requests sometimes and it does not trigger 429 or something.
Activity
max-wittig commentedon Feb 27, 2020
This is not the only flaky test in these functional tests. Many of them fail randomly, but just in CI, never locally.
I spend a lot of effort a while ago (even switched CI systems), but haven't figured it out, yet
I would be happy for every hint I get 😄
Or maybe we got to completely change this functional test setup, not sure.
nejch commentedon Feb 29, 2020
Ah I see :( I do get some failures locally, maybe my measly laptop is closer to Travis VM specs :P
It's hard to mark things for re-run now because certain tests prepare state for others, so maybe a refactor will help, let's see :)
[-]Functional test for rate limits is flaky[/-][+]Functional tests that create/update/delete resources are flaky[/+]nejch commentedon Mar 1, 2020
I now think that a lot of these failures happen when GitLab hands off tasks to Sidekiq asynchronously, and getting a response from the server does not mean that something is actually done (like deleting a project etc). If the next step/assert relies on that, it fails.
I see this more often now as I try to convert asserts to test cases for #1024 (I will push that once it gets greener :P ). Especially as I try to delete a resource in teardown and re-create it for the next test. If I run tests with
pytest-random-orderit goes green whent those 2 tests are further apart.For example with the rate limit, if I try to reuse
tools/reset_gitlab.pyfor teardown and add some checks withprojects.list(), it takes almost 10-20 seconds in my case to delete all the projects that the test creates (around 20-40 projects?).Have you or anyone here had any similar experience? Maybe we could have some kind of
wait_resource_available()andwait_resource_deleted()helpers, and have as much of that in fixtures as possible to avoid having sleeps in tests. It would probably slow down the tests quite a bit, but make them more robust. Anyway, splitting these into testcases is already some progress it seems :)