Skip to content

aamtests: Flaky issues on stability CI check due to enabling a11y in Chromium #62901

Open

Description

@mrego

In the last PRs adding aamtests, we have seen wpt-chrome-dev-stability complaining that 1 out of 10 runs is a TIMEOUT. That's quite consistent in CI runs, though I haven't managed to reproduce locally, even recreating the container.

After investigating this it looks the problem is that aamtests enable a11y, and sometimes CI has issues with that.

See #62888 for the simple examples and how CI has showed the problems in the following lines:

 0:51.20 TEST_START: /core-aam/aamtests/debug.py
 0:51.86 INFO STDOUT: ============================= test session starts ==============================
 0:51.86 INFO STDOUT: platform linux -- Python 3.10.12, pytest-8.2.1, pluggy-unknown -- /usr/bin/python3
 0:51.86 INFO STDOUT: rootdir: /home/test/web-platform-tests
 0:51.86 INFO STDOUT: plugins: asyncio-0.19.0
 0:51.86 INFO STDOUT: asyncio: mode=strict
 0:51.86 INFO STDOUT: collecting ... 
 0:51.90 INFO STDOUT: collected 1 item
 0:51.90 INFO STDOUT: core-aam/aamtests/debug.py::test_debug 
 0:51.91 INFO STDOUT: [session] - start
 0:51.91 INFO STDOUT: [Session.__init__] start
 0:51.91 INFO STDOUT: [Session.__init__] end
 0:51.91 INFO STDOUT: [session] - Call session.start()
 0:51.91 INFO STDOUT: [Session.start] start
 0:51.91 INFO STDOUT: [Session.start] call send_command()
 0:52.03 pid:2844 [2865:2865:0923/133832.013691:WARNING:components/discardable_memory/service/discardable_shared_memory_manager.cc:200] Less than 64MB of free space in temporary directory for shared memory files: 63
 1:21.21 TEST_END: TIMEOUT, expected OK
 1:21.21 INFO Restarting browser for new test group
 1:31.22 WARNING Forcibly terminating runner process
 1:31.24 INFO Starting WebDriver: /home/test/web-platform-tests/_venv3/bin/chrome/chromedriver --port=53281 --ignore-explicit-port --url-base=/ --enable-chrome-logs --disable-build-check
 1:31.26 INFO Got WebDriver port 42889
 1:31.36 pid:2888 Full command: /home/test/web-platform-tests/_venv3/bin/chrome/chromedriver --port=53281 --ignore-explicit-port --url-base=/ --enable-chrome-logs --disable-build-check
pid:2888 Starting ChromeDriver 156.0.8063.3 (aaa5c88cab4c6424c27f7ec3f607a9de4917f1da-refs/branch-heads/8063@{#6}) on port 0
 1:31.36 pid:2888 Only local connections are allowed.
 1:31.36 pid:2888 Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
 1:31.36 pid:2888 ChromeDriver was started successfully on port 42889.
 1:31.36 INFO Webdriver started successfully.
 1:31.36 INFO Starting runner
 1:31.67 TEST_START: /webdriver/tests/classic/get_title/debug.py
 1:31.84 INFO STDOUT: ============================= test session starts ==============================
 1:31.84 INFO STDOUT: platform linux -- Python 3.10.12, pytest-8.2.1, pluggy-unknown -- /usr/bin/python3
 1:31.84 INFO STDOUT: rootdir: /home/test/web-platform-tests/webdriver/tests
 1:31.84 INFO STDOUT: configfile: pytest.ini
 1:31.84 INFO STDOUT: plugins: asyncio-0.19.0
 1:31.84 INFO STDOUT: asyncio: mode=strict
 1:31.84 INFO STDOUT: collecting ... 
 1:31.86 INFO STDOUT: collected 1 item
 1:31.86 INFO STDOUT: webdriver/tests/classic/get_title/debug.py::test_debug 
 1:31.86 INFO STDOUT: [session] - start
 1:31.86 INFO STDOUT: [Session.__init__] start
 1:31.86 INFO STDOUT: [Session.__init__] end
 1:31.86 INFO STDOUT: [session] - Call session.start()
 1:31.86 INFO STDOUT: [Session.start] start
 1:31.86 INFO STDOUT: [Session.start] call send_command()
 1:31.89 pid:2888 [2909:2909:0923/133911.880854:WARNING:components/discardable_memory/service/discardable_shared_memory_manager.cc:200] Less than 64MB of free space in temporary directory for shared memory files: 63
 1:33.47 pid:2888 [2940:2940:0923/133913.455841:WARNING:media/gpu/vaapi/vaapi_wrapper.cc:1660] drmGetDevices2() has not found any devices
 1:33.47 pid:2888 [2940:2940:0923/133913.461499:WARNING:sandbox/policy/linux/sandbox_linux.cc:405] InitializeSandbox() called with multiple threads in process gpu-process.
 1:33.48 pid:2888 [2940:2940:0923/133913.466088:WARNING:ui/gfx/linux/gbm_support_x11.cc:48] dri3 extension not supported.
 1:33.76 pid:2888 [2909:2909:0923/133913.743671:WARNING:ui/aura/window.cc:1509] StackChildLayerRelativeTo called with unmanaged layer(s). child=WebContentsViewAura child_managed=0 target=WebContentsViewAura target_managed=0
 1:33.76 pid:2888 [2909:2909:0923/133913.743729:WARNING:ui/aura/window.cc:1509] StackChildLayerRelativeTo called with unmanaged layer(s). child=WebContentsViewAura child_managed=0 target=WebContentsViewAura target_managed=0
Failed to read: session.screen0.titlebar.left
Setting default value
Failed to read: session.screen0.titlebar.right
Setting default value
 1:33.79 pid:2888 [1790170753.773][WARNING]: You are using an unsupported command-line switch: --disable-build-check. Please don't report bugs that cannot be reproduced with this switch removed.
 1:33.88 pid:2888 [2909:2909:0923/133913.860360:ERROR:dbus/object_proxy.cc:572] Failed to call method: org.freedesktop.DBus.Properties.GetAll: object_path= /org/freedesktop/UPower/devices/DisplayDevice: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
 1:33.88 pid:2888 [2909:2909:0923/133913.860374:WARNING:dbus/property.cc:174] GetAll request failed for: org.freedesktop.UPower.Device
 1:33.91 pid:2888 [2909:2909:0923/133913.898156:WARNING:ui/base/idle/idle_linux.cc:111] None of the known D-Bus ScreenSaver services could be used.
 1:33.92 pid:2888 [2909:2909:0923/133913.904340:ERROR:dbus/object_proxy.cc:572] Failed to call method: org.freedesktop.NetworkManager.GetDevices: object_path= /org/freedesktop/NetworkManager: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.NetworkManager was not provided by any .service files
 1:33.92 pid:2888 [2909:2909:0923/133913.904354:WARNING:services/device/geolocation/wifi_data_provider_linux.cc:171] Failed to get the device list
 1:33.96 INFO STDOUT: [Session.start] send_command() returns
 1:33.96 INFO STDOUT: [Session.start] return value
 1:33.96 INFO STDOUT: [session] - session.start() returns
 1:34.40 INFO STDOUT: PASSED
 1:34.41 INFO STDOUT: [session] - end
 1:34.47 INFO STDOUT: ============================== 1 passed in 2.63s ===============================
 1:34.47 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0

The aamtest gets stuck in Session.start() for many seconds, while the other wdspec test has no issues. Other runs both tests are fast. The only difference is that aamtests enable a11y, see:

If you remove the ifs there and enable a11y unconditionally, the CI stability issue happens also for a wdspec test.

Activity

  1. mrego commented on Sep 24, 2026

    @mrego
    MemberAuthor

    Reported issue in Chromium bugtracker as it's not clear if it's a CI infra issue or something in Chromium a11y support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions