Skip to content

Rework the CI: light checks on every push, CPU tests in the merge queue, GPU tests on request - #3738

Merged
syurkevi merged 15 commits into
masterfrom
feature/ci-rework
Nov 3, 2026
Merged

syurkevi merged 15 commits into
masterfrom
feature/ci-rework

Conversation

@roaffix

@roaffix roaffix commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

This replaces ArrayFire's GitHub Actions CI, which can no longer run: the Linux jobs wait for the retired ubuntu-20.04 runners, and the Windows job fails within a minute. Every job is now one call to .github/scripts/ci.sh, which reproduces it locally. The CPU tests run in the merge queue and on request; the GPU tests (CUDA and OpenCL on GitHub's Tesla T4) run on request, after a maintainer's approval. No library code changes.

Description

Type: CI infrastructure, a new feature and a fix for the broken CI. Nothing changes under src/, include/, test/ or examples/, nor CMakeLists.txt, nor the vcpkg baseline or any dependency version.

What it does:

  • Light checks on every push and in the merge queue: lint (clang-format 15 on the changed lines), docs (Doxygen), and workflows (actionlint, shellcheck, the CI scripts' tests).

  • CPU tests (Linux GCC and Clang, macOS arm64, Windows MSVC), in the merge queue and on every push to master, and on a pull request when a maintainer comments /cpu-tests.

    • A merge whose changes are only non-code files (top-level Markdown, docs/, the templates) passes them in seconds and says why.
    • Any other change, and every request, runs them in full.
  • GPU tests, CUDA and OpenCL on GitHub's T4 runner, when a maintainer comments /gpu-tests.

    • They run only after one approval in the gpu-tests environment.
    • Without arguments, the PR's changed files choose CUDA, OpenCL or both.
    • Off until enabled.
  • Compile-only builds:

    • CUDA on Linux, for compute capability 7.5–12.0 including Blackwell;
    • CUDA on Windows;
    • OpenCL and oneAPI.

    For CUDA they also check the standalone packaging (which cuFFT, cuSOLVER and other libraries get bundled, under which names) and ArrayFire's default architecture list. They run on master and with /cpu-tests all. Off until enabled.

  • A nightly: Intel's compiler with MKL, the declared minimum CMake 3.16.3, and an early warning on the newest Ubuntu.

  • The Matrix Market (sparse) tests run again. SuiteSparse refuses CI's downloads (HTTP 403), so the 9 matrices come from a bundle checked against a SHA-256 manifest, from ci: restore build checks and CPU summation accuracy #3735.

  • The release workflow still builds the source tarball on tags. It now does so in a read-only job with a script, and uploads with gh in a separate job.

Why:

  • No CPU test has run on master for months.
  • The old jobs depended on things that are gone: retired runner labels, windows-latest changes, the CDash server, the SuiteSparse download.
  • ArrayFire had no GPU testing in CI.
  • The new jobs are pinned: container images, toolkit versions, and every action by commit SHA, with Dependabot. Each job runs the same way locally with ci.sh --docker <job>.

Impact on hardware, software and backends:

  • CPU: tested on Linux (GCC 13, Clang 18), macOS arm64 and Windows (MSVC 2022).
  • CUDA and OpenCL: tested on a Tesla T4 (sm_75) with CUDA 12.9 by default; CUDA 13.1 on request.
  • oneAPI and Windows CUDA: compile only.
  • GPU tests cost $0.052 per runner minute, about $0.50–0.80 per backend per request, and only after an approval. GitHub's GPU runners need the organisation on GitHub Team or Enterprise.

New functions: none.

Backporting: not needed; this changes CI only.

Related:

Not in this PR (possible follow-ups):

Changes to Users

  • Library users: nothing changes, with no new or changed build options.
  • CMake presets:
    • CMakePresets.json moves from format 2 to 6, so using the presets needs CMake 3.25 or newer. The project's own minimum stays 3.16.3, now tested nightly.
    • msvc2019-* is renamed msvc2022-*; no GitHub runner has Visual Studio 2019 anymore.
    • FFTW/LAPCK/BLAS is fixed to FFTW/LAPACK/BLAS in three presets. The typo made them skip their dependency checks.
    • New ci-* presets, one per CI job. The ninja-* presets are unchanged.
  • Contributors:
    • Any CI job runs locally with .github/scripts/ci.sh --docker <job> (--list shows them).
    • The CPU tests run in the merge queue, or earlier on request; see "Running CI" in CONTRIBUTING.md.
    • Open pull requests should use "Update branch" to pick up the new presets and the Windows vcpkg overlay. The request reply says when a branch is behind master.
  • Maintainers: the GPU tests and the compile-only builds stay off until an admin sets the variables and the environment (see "Settings" below).

Checklist

  • Rebased on latest master (ab4a5063c)
  • Code compiles: all CI jobs, including the CUDA, Windows CUDA, OpenCL and oneAPI compile jobs, pass in a test copy of the repository
  • Tests pass: CPU on all four platforms, plus CUDA and OpenCL on a T4 (see the evidence below)
  • Functions added to unified API: not applicable, no API change
  • Functions documented: not applicable. The CI is documented in CONTRIBUTING.md
Testing evidence

All of this ran in a test copy of the repository with these workflows, and locally with ci.sh --docker.

  • This branch, pushed as the test copy's master:
    • lint, docs and workflows passed;
    • the four CPU jobs passed;
    • windows-cuda (12.9) passed;
    • the CUDA, OpenCL and oneAPI compile jobs passed.
  • CPU test programs:
    • Linux GCC and Clang: 130 of 130, or 131 of 131 with the Matrix Market data (test_matrixmarket_cpu);
    • macOS: 130 of 130;
    • Windows: 129 of 129.
  • GPU tests on the T4, each request with one approval:
  • The T4 runner:
    • the "NVIDIA GPU-Optimized Image for AI and HPC", with driver 590.48.01 (CUDA 13.1);
    • CUDA 12.9 and 13.1 run natively;
    • newer toolkits get NVIDIA's forward-compatibility package;
    • OpenCL works once the job registers NVIDIA's ICD file, which the image lacks.
  • Compile jobs:
  • The nightly:
  • Request comments: the reply, the commit status, the result comment, and the "behind master" line all work. A replaced request is reported as cancelled, at no cost.
  • Not exercised yet on GitHub:
    • The merge queue itself, and its skip for non-code changes: it needs the ruleset, which is set after this merges. The skip's logic has 32 unit tests, and the first README-only pull request through the queue will show it.
    • Windows CUDA 13.x: the installer now adds the crt and nvvm components that CUDA 13 split out of nvcc.

Findings for the library, shared separately:

Jobs and when they run
Workflow Checks When
checks.yml lint, docs, workflows every push to a PR or master; the merge queue
cpu.yml linux-gcc-openblas, linux-clang-openblas, macos-arm64-clang-accelerate, windows-msvc-openblas (and a changes job that decides whether they build) the merge queue; pushes to master; /cpu-tests
cpu.yml accelerators (linux-cuda, <version>), accelerators (linux-opencl), accelerators (linux-oneapi), windows-cuda (<version>) pushes to master and /cpu-tests all, only with AF_CI_ACCELERATOR_BUILDS=true
cpu-placeholders.yml the four CPU check names, passing at once every PR push, so a PR can join a merge queue that requires them
cpu-nightly.yml linux-icx-mkl, linux-gcc-openblas-floor, linux-gcc-openblas-next daily at 03:00 UTC, in arrayfire/arrayfire only
gpu.yml build (…), approve, test (…), summary /gpu-tests, only with AF_CI_GPU_TESTS=true
test-requests.yml decide, reply, accepted, report a /cpu-tests or /gpu-tests comment
release_src_artifact.yml Upload release source tarball a release tag, as before
  • Security:
    • Pull-request code never runs with write permissions or secrets. test-requests.yml runs from master and builds the pinned commit with master's scripts.
    • Only master's own builds save caches.
Requesting tests on a pull request
Comment Who What runs
/cpu-tests maintainers, on any PR; a PR's author, on their own PR the four CPU jobs (macOS and Windows only for maintainers, since they build outside a container)
/cpu-tests all the same plus the compile-only builds, if enabled
/gpu-tests maintainers CUDA, OpenCL or both, chosen by the changed files; nothing for docs-only changes
/gpu-tests cuda, opencl, all, cuda=13.1, all cuda=13.1 maintainers the named backends; cuda=X.Y picks one CUDA version
  • The reply says what runs and why, and whether the branch is behind master.
  • The result is a cpu-tests or gpu-tests status and a comment.
  • One approval covers every GPU leg of a request, and can be given while the free builds run.
Settings to enable after merging, and how to switch each part off
Setting Value Off by
Ruleset on master required checks lint, docs, workflows, linux-gcc-openblas, linux-clang-openblas, macos-arm64-clang-accelerate, windows-msvc-openblas; merge queue disabling the ruleset
Variable AF_CI_ACCELERATOR_BUILDS true deleting it
Variable AF_CI_CUDA_VERSIONS 12.9 (add 13.1 once master passes the CUDA 13 checks) deleting it (12.9 is the default)
Variable AF_CI_GPU_RUNNER the T4 runner's name deleting it
Variable AF_CI_GPU_TESTS true, set last deleting it
Environment gpu-tests required reviewers: the maintainers' team; deployments from master —
T4 runner group public repositories allowed; restricted to test-requests.yml and gpu.yml removing the runner
"Always suggest updating pull request branches" on unticking it
  • The non-code skip can be narrowed in NON_CODE in .github/scripts/code-changes.py.
  • Everything can be undone by reverting the merge commit, which brings the old workflows back as they were.
Commits

Grouped by area; each passes ci.sh --docker workflows on its own.

  1. Replace the old CI workflows with ci.sh and its scripts
  2. Fix the presets' LAPACK value, the clFFT pin, Doxygen and .gitignore
  3. Add CMake presets for the CI configurations
  4. Add the light checks: lint, docs and workflows
  5. Take the LAPACK overlay and triplet from the fix/restore-ci branch
  6. Test the sparse functions with bundled Matrix Market data
  7. Add the CUDA release checks and the Windows CUDA scripts
  8. Add a GPU test workflow with one approval per request
  9. Run the CPU tests in the merge queue, on master and nightly
  10. Run CPU and GPU tests on request with /cpu-tests and /gpu-tests
  11. Describe CI in CONTRIBUTING.md
  12. Let Dependabot keep the pinned actions current

@roaffix
roaffix requested a review from syurkevi November 2, 2026 16:08
@roaffix roaffix self-assigned this Nov 2, 2026
@roaffix roaffix changed the title Feature/ci rework Rework the CI: light checks on every push, CPU tests in the merge queue, GPU tests on request Nov 2, 2026
@graehl

graehl commented Nov 3, 2026

Copy link
Copy Markdown

seems like this is blocking cuda 13 / blackwell (or at least tests will pass after this merges?
see failure: #3715
we'd really like to run afcuda on blackwell+ and although i can hack it myself, isn't it worth having officially merged?

@syurkevi

syurkevi commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

seems like this is blocking cuda 13 / blackwell (or at least tests will pass after this merges? see failure: #3715 we'd really like to run afcuda on blackwell+ and although i can hack it myself, isn't it worth having officially merged?

I'm testing cuda 13 builds right now. Hoping to have a PR opened and merged shortly 🤞

The Unix and Windows workflows downloaded their own CMake, Ninja and
Doxygen, ran partly on a retired runner image (Ubuntu 20.04), repeated
their build steps in YAML and reported to a CDash dashboard instead of
the pull request. They are removed here; their
jobs come back in the following commits as calls to one script. Doing
both in one commit keeps every workflow in the tree passing actionlint.

.github/scripts/ci.sh runs one CI job: it installs the job's
dependencies, then runs the job's CMake workflow preset or check, in
the job's pinned Ubuntu container (--docker) or on the macOS and
Windows hosts, so CI and a developer run the same command; --list
shows the jobs. With --docker it chooses the build parallelism from the
container's memory, keeps ccache in a volume or in a host directory CI
can cache, and builds either this checkout or another commit's tree
(AF_CI_SOURCE_DIR) with this repository's scripts and settings, also as
a copy without .git. It must run on macOS's bash 3.2.

install-deps-ubuntu.sh, install-deps-macos.sh and
install-deps-windows.ps1 install each job's packages: the compilers and
BLAS, the CUDA, OpenCL and oneAPI toolkits by version (with
nvJitLink's development package, which the installers' library bundling
looks up by name), the runtime libraries alone for GPU test containers,
and vcpkg at a pinned release (2026.10.16), whose MSYS2 helpers still
find their packages. lint.sh checks the formatting of the changed lines
with clang-format 15. checkout-commit.sh fetches one commit
anonymously. count-vcpkg-archives.sh tells whether a run added vcpkg
archives worth saving. .gitattributes keeps the shell scripts' LF
endings, which Git Bash on Windows runners needs.

The release source workflow builds the tarball with
make-source-tarball.sh and uploads it with gh release upload instead of
the archived upload-release-asset action; only the upload step can
write, and a manual run builds the tarball as an artifact without
publishing it. It no longer fetches the SuiteSparse matrices, which the
tarball never included.
- The non-MKL presets set AF_COMPUTE_LIBRARY to "FFTW/LAPCK/BLAS",
  which matches neither supported value, so configure skipped the
  FFTW, CBLAS and LAPACK checks.
- clFFT followed a moving branch, the only way the OpenCL build could
  change without a commit here. It is pinned to the commit its
  arrayfire-release branch points to, like every other OpenCL
  dependency.
- NUM_PROC_THREADS = 0 runs Doxygen's experimental parallel input
  processing on every core. With Doxygen 1.9.7 it intermittently
  corrupted the heap and aborted: 4 of 10 runs on a 24-core machine.
  On one thread every run passed, in the same time.
- The docs build writes docs/details/examples.dox into the source tree
  on every configure, which left the tree dirty; it is ignored.
The ci-* configure, build, test and workflow presets hold each CI
job's settings, so ci.sh runs "cmake --workflow --preset ci-<job>" and
a developer can run the same:

- CPU on Linux with GCC and Clang and OpenBLAS, with icx and oneMKL,
  and on the newest Ubuntu; macOS with Apple Clang and Accelerate,
  built with spdlog's bundled fmt (Homebrew's fmt 12 breaks it) and
  without the mean/var cases that FMA contraction breaks there; Windows
  with MSVC 2022 and vcpkg, at /W3 (/Wall flooded the log with
  thousands of system-header warnings) and with MSBuild's multi-tool
  task compiling in parallel;
- compile-only CUDA, OpenCL and oneAPI builds. CUDA builds for compute
  capabilities 7.5, 8.9 and 9.0, plus Blackwell's 10.0 and 12.0 through
  nvcc flags, because CUDA_architecture_build_targets goes through
  CMake's own FindCUDA copy of cuda_select_nvcc_arch_flags on this
  tree, which rejects two-digit majors; and a Windows CUDA build;
- the documentation.

The test presets write JUnit, cap the BLAS threads at four and fail
when no test runs. The Matrix Market tests follow AF_CI_MTX_FILES,
with FetchContent pointed at the data that ci.sh downloads from
arrayfire-data and restores into the build tree (added in a
following commit).

The Visual Studio 2019 presets become 2022 ones, since no GitHub runner
has 2019 any more, and the file moves to format 6 (CMake 3.25) for the
workflow presets.
checks.yml runs on every push to a pull request or master and in the
merge queue. It is cheap, so the CPU builds can run on a schedule of
their own:

- lint: clang-format 15 on the lines changed since the base. The tree
  has unformatted files, and library code isn't reformatted here.
- docs: the Doxygen build.
- workflows: actionlint (which also runs shellcheck on the workflows'
  run: steps), shellcheck on the CI scripts, the CI scripts' Python
  test suites, and check-bash32.sh.

check-bash32.sh runs the scripts that the macOS job runs on the host
under bash 3.2, the macOS runner's bash, in a bash:3.2 image pinned by
digest and without network: their syntax, ci.sh --list, a short SHA
through checkout-commit.sh, and a grep for bash-4-only syntax. Nothing
else caught bash 4 syntax until the macOS job failed.
vcpkg's current blas and lapack ports collide with the lapack-reference
overlay port: the overlay's lapack.pc, its blas.pc links for the
noblas feature and its wrapper in share/lapack belong to them now, and
the Windows build stopped at the file conflict. ArrayFire's own CI
pull request, the fix/restore-ci branch, fixes the overlay, and its
Windows run passes with it. Its portfile and triplet are taken as they
are, so the two branches don't conflict:

- the overlay installs lapack-reference.pc and its wrapper in
  share/lapack-reference, and no blas.pc links;
- vcpkg_cmake_config_fixup uses PACKAGE_NAME ${PORT};
- the triplet sets VCPKG_PROVIDED_FORTRAN, since vcpkg 2026.07.29 and
  later install MinGW gfortran for the port only when the triplet asks
  for it, as vcpkg's own x64-windows triplet does.
The sparse tests' matrices come from the SuiteSparse collection, whose
storage refuses every download ("403 AccessDenied"). ArrayFire's
arrayfire-data repository holds the same nine files, unchanged, under
matrixmarket/, with their CC BY 4.0 attribution. CI downloads them from
arrayfire-data at commit 2723164, which matrixmarket.json pins, so the
Matrix Market tests run again; an empty commit would turn them off.

The manifest of each matrix's size and SHA-256 and the idea of checking
the data before a build come from ArrayFire's fix/restore-ci branch
(#3735), whose check_matrix_data.py is adapted as check-matrix-data.py.
Before every build, ci.sh runs it. It downloads each matrix from the
pinned commit on raw.githubusercontent.com, checks it against the
manifest and the tested tree's list of matrices, fails naming every
file that differs or doesn't download, restores the matrices into the
build tree and prints ON; without a pinned commit it prints OFF.

ci.sh passes the answer on as AF_CI_MTX_FILES. The tests download each
matrix through af_dep_check_and_populate, whose FetchContent_Populate
takes FETCHCONTENT_SOURCE_DIR_<name> as the source when it is set; the
ci preset and matrixmarket.cmake (for the GPU builds and the CMake 3.16
build) set those to the restored matrices, and AF_TEST_WITH_MTX_FILES
to AF_CI_MTX_FILES. Nothing is downloaded from SuiteSparse, and nothing
under test/ changes. The GPU build archives carry the matrices to the
test jobs.
check-cuda-release.sh configures the CUDA backend as users and the
installers get it: ArrayFire's default architecture list and
AF_INSTALL_STANDALONE=ON. It prints the architecture flags and
compiles one CUDA object with them, which catches a default list with
an architecture the toolkit dropped (with CUDA 13, this tree's list
still has compute_53). It installs the cuda_dependencies component and
checks that every library the installers bundle on Linux is there,
under the toolkit's soname, the name libafcuda loads it by (with CUDA
13, this tree bundles cuFFT, cuSOLVER and cuSPARSE under the wrong
major). It needs only a configure and one compile.

install-cuda-windows.ps1 installs a CUDA toolkit on Windows with
NVIDIA's network installer, pinned by URL and SHA-256 per version
(12.9, 13.1, 13.4), with only the subpackages the build needs. From
CUDA 13.0 on, nvcc no longer includes crt (crt/host_config.h) and nvvm
(cicc): NVIDIA's redistrib manifests list them apart and the
installers carry crt_<version> and nvvm_<version>, though the
installation guides list them only from 13.2 on. msvc-env-windows.ps1
sets up Visual Studio's x64 developer environment for the Ninja
generator.
gpu.yml tests one commit on GPU runners. The scripts and settings come
from the ref it runs on, and only the source tree from the commit.
Each leg, a CUDA version from AF_CI_CUDA_VERSIONS or OpenCL, is built
on a free standard runner into an archive of the test programs,
libraries and test data, then tested on a GPU runner (AF_CI_GPU_RUNNER)
in a container with the runtime libraries alone, which installs in
about a third of the time of the toolkit. One approve job holds the
gpu-tests environment, so one reviewer's approval covers every leg, and
it starts with the builds. A failed build fails only its own leg; a
summary job reports every leg's passed and failed tests.

The CUDA test job compares the driver with the toolkit (cuda-compat.sh)
and, when the driver is older, installs NVIDIA's forward-compatibility
package for data-center GPUs such as the T4: ArrayFire's JIT has the
driver load PTX from NVRTC, which minor-version compatibility doesn't
cover. The OpenCL test job registers NVIDIA's ICD when the container
toolkit doesn't, prints the platforms and fails clearly without an
NVIDIA one. linux-cuda-gpu.cmake and linux-opencl-gpu.cmake hold the
builds' trusted settings for any commit.

The workflow is off unless AF_CI_GPU_TESTS is true.
cpu.yml runs the CPU builds and tests on Linux (GCC and Clang, in
pinned containers), macOS and Windows, in the merge queue and on every
push to master, and for a given commit when started by hand or by
another workflow; such a commit is built on its own with the trusted
scripts and presets. Actions are pinned to commit SHAs. Only master's
own builds save the ccache, the vcpkg cache and the downloaded Matrix
Market data, because a build of another commit could plant entries in
them; the vcpkg cache is saved
whenever a run added archives, even if the build failed later. The
macOS and Windows jobs build outside a container, so a request runs
them only when its caller asks for them.

With AF_CI_ACCELERATOR_BUILDS set to true, pushes and requests also
compile the CUDA (once per AF_CI_CUDA_VERSIONS version, with Blackwell
targets and the release checks), OpenCL and oneAPI backends, and the
CUDA backend on Windows. The matrix job has a fixed name: GitHub never
expands a skipped matrix job's name, which showed up as a check named
"matrix.job".

cpu-placeholders.yml gives a pull request instant checks with the same
names, so that it can join a merge queue that requires them. The
nightly runs the icx and oneMKL job, the job on the minimum CMake
version and the job on the newest Ubuntu, which may fail without
failing the run.

A merge-queue entry or a push to master that only adds or changes
non-code files (code-changes.py: Markdown at the top level, docs/, the
issue and pull-request templates, GitHub's citation files, .gitignore,
.mailmap) builds nothing. The changes job lists the changed files with
the compare API and the read-only token; a merge-queue entry is
compared with the branch it merges into, since the group's parent may
be the entry ahead. None of those files is read by a CPU build, an
installer rule or the release tarball. A removed or renamed file runs
the tests in full, since CPack checks at every Linux configure that
README.md exists; so does any other file and every request. The CPU
jobs still run and skip their steps, so the required checks keep their
names (a skipped matrix job's name isn't expanded) and pass in seconds,
each saying why.
A /cpu-tests or /gpu-tests comment on a pull request asks for the
tests of its current head. test-requests.yml runs from the default
branch, so the pull request can't change it, and nothing from the pull
request is checked out or run in it. test-requests.py decides, pins
the head SHA, and cpu.yml or gpu.yml test exactly that commit; the
result goes to the cpu-tests or gpu-tests commit status and a comment.

- /cpu-tests (all adds the accelerator builds): maintainers with write
  access on any pull request; the pull request's author on their own,
  once the light checks have run on it, on Linux only.
- /gpu-tests [cuda|opencl|all] [cuda=X.Y]: maintainers with write
  access only. Without a backend the changed files choose, and a
  change that can't affect a GPU backend gets a reply and starts
  nothing; CUDA versions come from AF_CI_CUDA_VERSIONS (12.9 when
  unset). Every reply says what runs and why.
- An accepted reply says when the branch is behind its base, since CI
  fixes made there since aren't in the test.

Failed API calls become annotations of the run, readable without
logging in; a failed comment after the status is set is only a
warning. The tests run the decisions on fixtures, the commands with
failing writes, and the CUDA driver check.
The "Running CI" section says what runs when, which checks the light
checks are, how to request the CPU and GPU tests and what they cost,
how to run any CI job locally with ci.sh, and where the Matrix Market
data comes from.
The workflows pin every action to a commit SHA; Dependabot proposes the
updates weekly.
@@ -0,0 +1,87 @@
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AF_TEST_WITH_MTX_FILES=OFF

@syurkevi
syurkevi merged commit 33719e5 into master Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants