Skip to content

Commit d0e6d8e

Browse files
GarbeeGarbee
authored
ci(nightly): port testing to github actions (#4918)
This patch begins moving the workflows from CircleCI to Github. In order to minimize the reviews, the first chunk is only converting the nightly tests. # Key Changes ## Removal of Circle CI Config for Nightly Tests Since this patch adds support for running nightly tests in GitHub Actions, we no longer need it in Circle CI. It would be wasteful of resources to do the same work in multiple places. ## Install Deps Composite Action This file reduces duplication of steps between multiple jobs and workflows. It is called after a repository is checked out in CI in order to install core OS dependencies, browsers, package dependencies, and starts (when needed) a virtual X server for running headed browsers against for testing. ## Nightly Tests The nightly workflow runs the current beta releases for Firefox and Chrome, WCAG ACT's current version, and the latest aria practices tests. ## Webdriver Configuration Previously, `browser-driver-manager` was used to force install the correct browser, driver, and updated selenium to match. This caused specific quirks in ordering of steps that had to be adhered to. With this conversion, the sources are now setup to take in the needed paths by an environment variable. This allows the `install-deps` action to install the browsers as they are an OS level dependency to run. Then the paths are passed to the test runs and utilized. Bypassing the need to run certain installs in a specific order. # Verification The original code in the PR was running on every push to get a test run going. [This run](https://github.com/dequelabs/axe-core/actions/runs/18976324783) was from the commit before the final push which turned off the `on: push` execution. This shows the workflow successfully running with all nightly tests passing. Refs: #4912
1 parent 671de01 commit d0e6d8e

File tree

6 files changed

+177
-94
lines changed

6 files changed

+177
-94
lines changed

‎.circleci/config.yml‎

Lines changed: 0 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,6 @@ unix_box: &unix_box
77
docker:
88
- image: cimg/node:22.16-browsers
99

10-
unix_nightly_box: &unix_nightly_box
11-
docker:
12-
- image: cimg/node:lts-browsers
13-
1410
orbs:
1511
puppeteer: threetreeslight/puppeteer@0.1.2
1612
browser-tools: circleci/browser-tools@1.5.1
@@ -169,61 +165,6 @@ jobs:
169165
- <<: *restore_build
170166
- run: npm run test:virtual-rules
171167

172-
# Run the test suite for nightly builds.
173-
test_nightly_browsers:
174-
<<: *defaults
175-
<<: *unix_nightly_box
176-
steps:
177-
- checkout
178-
- <<: *restore_dependency_cache_unix
179-
- run: sudo apt-get update -y
180-
- <<: *restore_build
181-
- run:
182-
name: Install Chrome and ChromeDriver Beta
183-
command: npx browser-driver-manager install chrome=beta chromedriver=beta --verbose
184-
- run:
185-
name: Install Firefox Nightly
186-
command: |
187-
# Assumes Firefox >= 135; in earlier versions, this resolves to a .tar.bz2 instead
188-
wget -O firefox-nightly.tar.xz "https://download.mozilla.org/?product=firefox-nightly-latest-ssl&os=linux64&lang=en-US"
189-
tar xf firefox-nightly.tar.xz
190-
- run:
191-
name: Set Environment Variable
192-
command: echo "export FIREFOX_NIGHTLY_BIN=$(pwd)/firefox/firefox-bin" >> $BASH_ENV
193-
- run: npm run test -- --browsers Chrome,FirefoxNightly
194-
195-
# Run the test suite for nightly builds.
196-
test_nightly_act:
197-
<<: *defaults
198-
<<: *unix_nightly_box
199-
steps:
200-
- checkout
201-
- <<: *restore_dependency_cache_unix
202-
- browser-tools-job
203-
# install ACT rules
204-
# install first as for some reason installing a single package
205-
# also re-installs all repo dependencies as well
206-
- run: npm install w3c/wcag-act-rules#main
207-
- run: npx browser-driver-manager install chromedriver --verbose
208-
- <<: *restore_build
209-
- run: npm run test:act
210-
211-
# Run the test suite for nightly builds.
212-
test_nightly_aria_practices:
213-
<<: *defaults
214-
<<: *unix_nightly_box
215-
steps:
216-
- checkout
217-
- <<: *restore_dependency_cache_unix
218-
- browser-tools-job
219-
# install ARIA practices
220-
# install first as for some reason installing a single package
221-
# also re-installs all repo dependencies as well
222-
- run: npm install w3c/aria-practices#main
223-
- run: npx browser-driver-manager install chromedriver --verbose
224-
- <<: *restore_build
225-
- run: npm run test:apg
226-
227168
# Test api docs can be built
228169
build_api_docs:
229170
<<: *defaults
@@ -433,26 +374,3 @@ workflows:
433374
- github_release:
434375
requires:
435376
- release
436-
nightly:
437-
triggers:
438-
- schedule:
439-
# run at 00:00 UTC every day
440-
cron: '0 0 * * *'
441-
filters:
442-
branches:
443-
only:
444-
- develop
445-
jobs:
446-
- dependencies_unix
447-
- build_unix:
448-
requires:
449-
- dependencies_unix
450-
- test_nightly_browsers:
451-
requires:
452-
- build_unix
453-
- test_nightly_act:
454-
requires:
455-
- build_unix
456-
- test_nightly_aria_practices:
457-
requires:
458-
- build_unix

‎.github/actions/install-deps/action.yml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
name: 'Install Dependencies'
2+
description: 'Install OS and Project dependencies'
3+
4+
inputs:
5+
node-version:
6+
description: 'Node.js version to install'
7+
required: false
8+
start-xvfb:
9+
description: 'If provided, this is the display number to run xvfb on. Should be in `:N` format, e.g., `:99`.'
10+
required: false
11+
nightly:
12+
description: 'If true, installs the nightly versions of browsers.'
13+
required: false
14+
outputs:
15+
chrome-path:
16+
description: 'Path to the installed Chrome binary'
17+
value: ${{ steps.setup-chrome.outputs.chrome-path }}
18+
firefox-path:
19+
description: 'Path to the installed Firefox binary'
20+
value: ${{ steps.setup-firefox.outputs.firefox-path }}
21+
chromedriver-path:
22+
description: 'Path to the installed ChromeDriver binary'
23+
value: ${{ steps.setup-chrome.outputs.chromedriver-path }}
24+
chrome-version:
25+
description: 'Version of the installed Chrome binary'
26+
value: ${{ steps.setup-chrome.outputs.chrome-version }}
27+
chromedriver-version:
28+
description: 'Version of the installed ChromeDriver binary'
29+
value: ${{ steps.setup-chrome.outputs.chromedriver-version }}
30+
firefox-version:
31+
description: 'Version of the installed Firefox binary'
32+
value: ${{ steps.setup-firefox.outputs.firefox-version }}
33+
34+
runs:
35+
using: 'composite'
36+
steps:
37+
- uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
38+
with:
39+
registry-url: 'https://registry.npmjs.org'
40+
node-version: ${{ inputs.node-version }}
41+
node-version-file: ${{ inputs.node-version == '' && '.nvmrc' || '' }}
42+
cache: npm
43+
- name: Fix Chrome Sandbox Permissions
44+
shell: bash
45+
run: |
46+
sudo chown root:root /opt/google/chrome/chrome-sandbox
47+
sudo chmod 4755 /opt/google/chrome/chrome-sandbox
48+
- name: Install Xvfb
49+
shell: bash
50+
if: ${{ inputs.start-xvfb }}
51+
run: |
52+
sudo apt-get update
53+
sudo apt-get install -y xvfb x11-xserver-utils
54+
- name: Install Google Chrome for Testing
55+
id: setup-chrome
56+
uses: browser-actions/setup-chrome@b94431e051d1c52dcbe9a7092a4f10f827795416 # v2.1.0
57+
with:
58+
chrome-version: ${{ inputs.nightly == 'true' && 'beta' || 'stable' }}
59+
install-chromedriver: true
60+
install-dependencies: true
61+
- name: Install Firefox
62+
id: setup-firefox
63+
uses: browser-actions/setup-firefox@5914774dda97099441f02628f8d46411fcfbd208 # v1.7.0
64+
with:
65+
firefox-version: ${{ inputs.nightly == 'true' && 'latest-nightly' || 'latest' }}
66+
- name: Install Project Dependencies
67+
shell: bash
68+
run: npm ci
69+
- name: Start Xvfb
70+
if: ${{ inputs.start-xvfb }}
71+
shell: bash
72+
# This is the same resolution as what CircleCI used.
73+
# Maintaining it for consistency between the environments
74+
# since something may be resolution dependent.
75+
run: Xvfb ${{ inputs.start-xvfb }} -screen 0 1280x1024x24 &

‎.github/workflows/nightly-tests.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Nightly Tests
2+
3+
on:
4+
schedule:
5+
# Runs every day at 2:17 AM UTC
6+
# Schedules should try to be offset from common times
7+
# to avoid high contention times on GitHub runners.
8+
- cron: '17 2 * * *'
9+
workflow_dispatch:
10+
11+
env:
12+
CHROME_DEVEL_SANDBOX: /opt/google/chrome/chrome-sandbox
13+
14+
permissions:
15+
contents: read
16+
17+
jobs:
18+
browsers:
19+
runs-on: ubuntu-24.04
20+
timeout-minutes: 10
21+
env:
22+
DISPLAY: ':99'
23+
steps:
24+
- &checkout
25+
name: Checkout repository
26+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
27+
- name: Install Dependencies
28+
id: install-deps
29+
uses: ./.github/actions/install-deps
30+
with:
31+
nightly: 'true'
32+
start-xvfb: ${{ env.DISPLAY }}
33+
- &build
34+
name: Build
35+
run: |
36+
npm run prepare
37+
npm run build
38+
- name: Run Firefox Nightly Browser Tests
39+
env:
40+
FIREFOX_NIGHTLY_BIN: ${{ steps.install-deps.outputs.firefox-path }}
41+
run: npm run test -- --browsers FirefoxNightly
42+
- name: Run Chrome Beta Browser Tests
43+
env:
44+
CHROME_BIN: ${{ steps.install-deps.outputs.chrome-path }}
45+
CHROMEDRIVER_BIN: ${{ steps.install-deps.outputs.chromedriver-path }}
46+
run: npm run test -- --browsers Chrome
47+
act:
48+
runs-on: ubuntu-24.04
49+
timeout-minutes: 10
50+
steps:
51+
- *checkout
52+
- &install-deps
53+
name: Install Deps
54+
id: install-deps
55+
uses: ./.github/actions/install-deps
56+
- *build
57+
- name: Install Latest WCAG ACT Rules
58+
run: npm install w3c/wcag-act-rules#main
59+
- name: Run ACT Tests
60+
env:
61+
CHROME_BIN: ${{ steps.install-deps.outputs.chrome-path }}
62+
CHROMEDRIVER_BIN: ${{ steps.install-deps.outputs.chromedriver-path }}
63+
run: npm run test:act
64+
aria-practices:
65+
runs-on: ubuntu-24.04
66+
timeout-minutes: 7
67+
steps:
68+
- *checkout
69+
- *install-deps
70+
- *build
71+
- name: Install Latest W3C Aria Practices
72+
run: npm install w3c/aria-practices#main
73+
- name: Run ARIA Practices Tests
74+
env:
75+
CHROME_BIN: ${{ steps.install-deps.outputs.chrome-path }}
76+
CHROMEDRIVER_BIN: ${{ steps.install-deps.outputs.chromedriver-path }}
77+
run: npm run test:apg

‎test/act-rules/act-runner.js‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,8 @@ module.exports = ({ id, title, axeRules, skipTests = [] }) => {
2828
this.timeout(50000);
2929
this.retries(3);
3030

31-
before(async () => {
32-
driver = getWebdriver();
33-
});
34-
3531
before(done => {
32+
driver = getWebdriver();
3633
server = http.createServer((request, response) => {
3734
return handler(request, response, {
3835
cleanUrls: false,

‎test/get-webdriver.js‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,21 @@
11
const { Builder } = require('selenium-webdriver');
22
const chrome = require('selenium-webdriver/chrome');
3-
const chromedriverPath = require('chromedriver').path;
3+
const chromedriverPath =
4+
process.env.CHROMEDRIVER_BIN ?? require('chromedriver').path;
45

56
const getWebdriver = () => {
67
const service = new chrome.ServiceBuilder(chromedriverPath);
8+
const options = new chrome.Options().addArguments('--headless');
79

8-
const webdriver = new Builder()
9-
.setChromeOptions(new chrome.Options().addArguments('headless'))
10+
if (process.env.CHROME_BIN) {
11+
options.setBinaryPath(process.env.CHROME_BIN);
12+
}
13+
14+
return new Builder()
15+
.setChromeOptions(options)
1016
.forBrowser('chrome')
1117
.setChromeService(service)
1218
.build();
13-
return webdriver;
1419
};
1520

1621
module.exports.getWebdriver = getWebdriver;

‎test/integration/full/test-webdriver.js‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
const { globSync } = require('glob');
22
const chrome = require('selenium-webdriver/chrome');
33
const firefox = require('selenium-webdriver/firefox');
4-
const chromedriver = require('chromedriver');
4+
const chromedriver =
5+
process.env.CHROMEDRIVER_BIN ?? require('chromedriver').path;
56

67
const args = process.argv.slice(2);
78

@@ -118,22 +119,32 @@ function buildWebDriver(browser) {
118119
let webdriver;
119120
const mobileBrowser = browser.split('-mobile');
120121

121-
// fix chrome DevToolsActivePort file doesn't exist in CricleCI (as well as a
122+
// fix chrome DevToolsActivePort file doesn't exist in CircleCI (as well as a
122123
// host of other problems with starting Chrome). the only thing that seems to
123124
// allow Chrome to start without problems consistently is using ChromeHeadless
124125
// @see https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t
125126
if (browser === 'chrome') {
126-
const service = new chrome.ServiceBuilder(chromedriver.path).build();
127+
const service = new chrome.ServiceBuilder(chromedriver).build();
127128

128129
const options = new chrome.Options().addArguments([
129-
'headless',
130+
'--headless',
130131
'--no-sandbox',
131132
'--disable-extensions',
132133
'--disable-dev-shm-usage'
133134
]);
135+
136+
if (process.env.CHROME_BIN) {
137+
options.setBinaryPath(process.env.CHROME_BIN);
138+
}
139+
134140
webdriver = chrome.Driver.createSession(options, service);
135141
} else if (browser === 'firefox') {
136142
const options = new firefox.Options().addArguments('--headless');
143+
144+
if (process.env.FIREFOX_BIN) {
145+
options.setBinaryPath(process.env.FIREFOX_BIN);
146+
}
147+
137148
webdriver = firefox.Driver.createSession(options);
138149
}
139150

0 commit comments

Comments
 (0)