Repository navigation
Rework the CI: light checks on every push, CPU tests in the merge queue, GPU tests on request - #3738
Merged
Merged
Conversation
5 tasks done
|
seems like this is blocking cuda 13 / blackwell (or at least tests will pass after this merges? |
Contributor
I'm testing cuda 13 builds right now. Hoping to have a PR opened and merged shortly 🤞 |
The Unix and Windows workflows downloaded their own CMake, Ninja and Doxygen, ran partly on a retired runner image (Ubuntu 20.04), repeated their build steps in YAML and reported to a CDash dashboard instead of the pull request. They are removed here; their jobs come back in the following commits as calls to one script. Doing both in one commit keeps every workflow in the tree passing actionlint. .github/scripts/ci.sh runs one CI job: it installs the job's dependencies, then runs the job's CMake workflow preset or check, in the job's pinned Ubuntu container (--docker) or on the macOS and Windows hosts, so CI and a developer run the same command; --list shows the jobs. With --docker it chooses the build parallelism from the container's memory, keeps ccache in a volume or in a host directory CI can cache, and builds either this checkout or another commit's tree (AF_CI_SOURCE_DIR) with this repository's scripts and settings, also as a copy without .git. It must run on macOS's bash 3.2. install-deps-ubuntu.sh, install-deps-macos.sh and install-deps-windows.ps1 install each job's packages: the compilers and BLAS, the CUDA, OpenCL and oneAPI toolkits by version (with nvJitLink's development package, which the installers' library bundling looks up by name), the runtime libraries alone for GPU test containers, and vcpkg at a pinned release (2026.10.16), whose MSYS2 helpers still find their packages. lint.sh checks the formatting of the changed lines with clang-format 15. checkout-commit.sh fetches one commit anonymously. count-vcpkg-archives.sh tells whether a run added vcpkg archives worth saving. .gitattributes keeps the shell scripts' LF endings, which Git Bash on Windows runners needs. The release source workflow builds the tarball with make-source-tarball.sh and uploads it with gh release upload instead of the archived upload-release-asset action; only the upload step can write, and a manual run builds the tarball as an artifact without publishing it. It no longer fetches the SuiteSparse matrices, which the tarball never included.
- The non-MKL presets set AF_COMPUTE_LIBRARY to "FFTW/LAPCK/BLAS", which matches neither supported value, so configure skipped the FFTW, CBLAS and LAPACK checks. - clFFT followed a moving branch, the only way the OpenCL build could change without a commit here. It is pinned to the commit its arrayfire-release branch points to, like every other OpenCL dependency. - NUM_PROC_THREADS = 0 runs Doxygen's experimental parallel input processing on every core. With Doxygen 1.9.7 it intermittently corrupted the heap and aborted: 4 of 10 runs on a 24-core machine. On one thread every run passed, in the same time. - The docs build writes docs/details/examples.dox into the source tree on every configure, which left the tree dirty; it is ignored.
The ci-* configure, build, test and workflow presets hold each CI job's settings, so ci.sh runs "cmake --workflow --preset ci-<job>" and a developer can run the same: - CPU on Linux with GCC and Clang and OpenBLAS, with icx and oneMKL, and on the newest Ubuntu; macOS with Apple Clang and Accelerate, built with spdlog's bundled fmt (Homebrew's fmt 12 breaks it) and without the mean/var cases that FMA contraction breaks there; Windows with MSVC 2022 and vcpkg, at /W3 (/Wall flooded the log with thousands of system-header warnings) and with MSBuild's multi-tool task compiling in parallel; - compile-only CUDA, OpenCL and oneAPI builds. CUDA builds for compute capabilities 7.5, 8.9 and 9.0, plus Blackwell's 10.0 and 12.0 through nvcc flags, because CUDA_architecture_build_targets goes through CMake's own FindCUDA copy of cuda_select_nvcc_arch_flags on this tree, which rejects two-digit majors; and a Windows CUDA build; - the documentation. The test presets write JUnit, cap the BLAS threads at four and fail when no test runs. The Matrix Market tests follow AF_CI_MTX_FILES, with FetchContent pointed at the data that ci.sh downloads from arrayfire-data and restores into the build tree (added in a following commit). The Visual Studio 2019 presets become 2022 ones, since no GitHub runner has 2019 any more, and the file moves to format 6 (CMake 3.25) for the workflow presets.
checks.yml runs on every push to a pull request or master and in the merge queue. It is cheap, so the CPU builds can run on a schedule of their own: - lint: clang-format 15 on the lines changed since the base. The tree has unformatted files, and library code isn't reformatted here. - docs: the Doxygen build. - workflows: actionlint (which also runs shellcheck on the workflows' run: steps), shellcheck on the CI scripts, the CI scripts' Python test suites, and check-bash32.sh. check-bash32.sh runs the scripts that the macOS job runs on the host under bash 3.2, the macOS runner's bash, in a bash:3.2 image pinned by digest and without network: their syntax, ci.sh --list, a short SHA through checkout-commit.sh, and a grep for bash-4-only syntax. Nothing else caught bash 4 syntax until the macOS job failed.
vcpkg's current blas and lapack ports collide with the lapack-reference
overlay port: the overlay's lapack.pc, its blas.pc links for the
noblas feature and its wrapper in share/lapack belong to them now, and
the Windows build stopped at the file conflict. ArrayFire's own CI
pull request, the fix/restore-ci branch, fixes the overlay, and its
Windows run passes with it. Its portfile and triplet are taken as they
are, so the two branches don't conflict:
- the overlay installs lapack-reference.pc and its wrapper in
share/lapack-reference, and no blas.pc links;
- vcpkg_cmake_config_fixup uses PACKAGE_NAME ${PORT};
- the triplet sets VCPKG_PROVIDED_FORTRAN, since vcpkg 2026.07.29 and
later install MinGW gfortran for the port only when the triplet asks
for it, as vcpkg's own x64-windows triplet does.
The sparse tests' matrices come from the SuiteSparse collection, whose
storage refuses every download ("403 AccessDenied"). ArrayFire's
arrayfire-data repository holds the same nine files, unchanged, under
matrixmarket/, with their CC BY 4.0 attribution. CI downloads them from
arrayfire-data at commit 2723164, which matrixmarket.json pins, so the
Matrix Market tests run again; an empty commit would turn them off.
The manifest of each matrix's size and SHA-256 and the idea of checking
the data before a build come from ArrayFire's fix/restore-ci branch
(#3735), whose check_matrix_data.py is adapted as check-matrix-data.py.
Before every build, ci.sh runs it. It downloads each matrix from the
pinned commit on raw.githubusercontent.com, checks it against the
manifest and the tested tree's list of matrices, fails naming every
file that differs or doesn't download, restores the matrices into the
build tree and prints ON; without a pinned commit it prints OFF.
ci.sh passes the answer on as AF_CI_MTX_FILES. The tests download each
matrix through af_dep_check_and_populate, whose FetchContent_Populate
takes FETCHCONTENT_SOURCE_DIR_<name> as the source when it is set; the
ci preset and matrixmarket.cmake (for the GPU builds and the CMake 3.16
build) set those to the restored matrices, and AF_TEST_WITH_MTX_FILES
to AF_CI_MTX_FILES. Nothing is downloaded from SuiteSparse, and nothing
under test/ changes. The GPU build archives carry the matrices to the
test jobs.
check-cuda-release.sh configures the CUDA backend as users and the installers get it: ArrayFire's default architecture list and AF_INSTALL_STANDALONE=ON. It prints the architecture flags and compiles one CUDA object with them, which catches a default list with an architecture the toolkit dropped (with CUDA 13, this tree's list still has compute_53). It installs the cuda_dependencies component and checks that every library the installers bundle on Linux is there, under the toolkit's soname, the name libafcuda loads it by (with CUDA 13, this tree bundles cuFFT, cuSOLVER and cuSPARSE under the wrong major). It needs only a configure and one compile. install-cuda-windows.ps1 installs a CUDA toolkit on Windows with NVIDIA's network installer, pinned by URL and SHA-256 per version (12.9, 13.1, 13.4), with only the subpackages the build needs. From CUDA 13.0 on, nvcc no longer includes crt (crt/host_config.h) and nvvm (cicc): NVIDIA's redistrib manifests list them apart and the installers carry crt_<version> and nvvm_<version>, though the installation guides list them only from 13.2 on. msvc-env-windows.ps1 sets up Visual Studio's x64 developer environment for the Ninja generator.
gpu.yml tests one commit on GPU runners. The scripts and settings come from the ref it runs on, and only the source tree from the commit. Each leg, a CUDA version from AF_CI_CUDA_VERSIONS or OpenCL, is built on a free standard runner into an archive of the test programs, libraries and test data, then tested on a GPU runner (AF_CI_GPU_RUNNER) in a container with the runtime libraries alone, which installs in about a third of the time of the toolkit. One approve job holds the gpu-tests environment, so one reviewer's approval covers every leg, and it starts with the builds. A failed build fails only its own leg; a summary job reports every leg's passed and failed tests. The CUDA test job compares the driver with the toolkit (cuda-compat.sh) and, when the driver is older, installs NVIDIA's forward-compatibility package for data-center GPUs such as the T4: ArrayFire's JIT has the driver load PTX from NVRTC, which minor-version compatibility doesn't cover. The OpenCL test job registers NVIDIA's ICD when the container toolkit doesn't, prints the platforms and fails clearly without an NVIDIA one. linux-cuda-gpu.cmake and linux-opencl-gpu.cmake hold the builds' trusted settings for any commit. The workflow is off unless AF_CI_GPU_TESTS is true.
cpu.yml runs the CPU builds and tests on Linux (GCC and Clang, in pinned containers), macOS and Windows, in the merge queue and on every push to master, and for a given commit when started by hand or by another workflow; such a commit is built on its own with the trusted scripts and presets. Actions are pinned to commit SHAs. Only master's own builds save the ccache, the vcpkg cache and the downloaded Matrix Market data, because a build of another commit could plant entries in them; the vcpkg cache is saved whenever a run added archives, even if the build failed later. The macOS and Windows jobs build outside a container, so a request runs them only when its caller asks for them. With AF_CI_ACCELERATOR_BUILDS set to true, pushes and requests also compile the CUDA (once per AF_CI_CUDA_VERSIONS version, with Blackwell targets and the release checks), OpenCL and oneAPI backends, and the CUDA backend on Windows. The matrix job has a fixed name: GitHub never expands a skipped matrix job's name, which showed up as a check named "matrix.job". cpu-placeholders.yml gives a pull request instant checks with the same names, so that it can join a merge queue that requires them. The nightly runs the icx and oneMKL job, the job on the minimum CMake version and the job on the newest Ubuntu, which may fail without failing the run. A merge-queue entry or a push to master that only adds or changes non-code files (code-changes.py: Markdown at the top level, docs/, the issue and pull-request templates, GitHub's citation files, .gitignore, .mailmap) builds nothing. The changes job lists the changed files with the compare API and the read-only token; a merge-queue entry is compared with the branch it merges into, since the group's parent may be the entry ahead. None of those files is read by a CPU build, an installer rule or the release tarball. A removed or renamed file runs the tests in full, since CPack checks at every Linux configure that README.md exists; so does any other file and every request. The CPU jobs still run and skip their steps, so the required checks keep their names (a skipped matrix job's name isn't expanded) and pass in seconds, each saying why.
A /cpu-tests or /gpu-tests comment on a pull request asks for the tests of its current head. test-requests.yml runs from the default branch, so the pull request can't change it, and nothing from the pull request is checked out or run in it. test-requests.py decides, pins the head SHA, and cpu.yml or gpu.yml test exactly that commit; the result goes to the cpu-tests or gpu-tests commit status and a comment. - /cpu-tests (all adds the accelerator builds): maintainers with write access on any pull request; the pull request's author on their own, once the light checks have run on it, on Linux only. - /gpu-tests [cuda|opencl|all] [cuda=X.Y]: maintainers with write access only. Without a backend the changed files choose, and a change that can't affect a GPU backend gets a reply and starts nothing; CUDA versions come from AF_CI_CUDA_VERSIONS (12.9 when unset). Every reply says what runs and why. - An accepted reply says when the branch is behind its base, since CI fixes made there since aren't in the test. Failed API calls become annotations of the run, readable without logging in; a failed comment after the status is set is only a warning. The tests run the decisions on fixtures, the commands with failing writes, and the CUDA driver check.
The "Running CI" section says what runs when, which checks the light checks are, how to request the CPU and GPU tests and what they cost, how to run any CI job locally with ci.sh, and where the Matrix Market data comes from.
The workflows pin every action to a commit SHA; Dependabot proposes the updates weekly.
roaffix
force-pushed
the
feature/ci-rework
branch
from
November 3, 2026 05:37
2d8a354 to
847d393
Compare
syurkevi
force-pushed
the
feature/ci-rework
branch
from
November 3, 2026 23:53
fa586db to
da4c616
Compare
syurkevi
approved these changes
Nov 3, 2026
| @@ -0,0 +1,87 @@ | |||
| { | |||
Contributor
There was a problem hiding this comment.
AF_TEST_WITH_MTX_FILES=OFF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This replaces ArrayFire's GitHub Actions CI, which can no longer run: the Linux jobs wait for the retired
ubuntu-20.04runners, and the Windows job fails within a minute. Every job is now one call to.github/scripts/ci.sh, which reproduces it locally. The CPU tests run in the merge queue and on request; the GPU tests (CUDA and OpenCL on GitHub's Tesla T4) run on request, after a maintainer's approval. No library code changes.Description
Type: CI infrastructure, a new feature and a fix for the broken CI. Nothing changes under
src/,include/,test/orexamples/, norCMakeLists.txt, nor the vcpkg baseline or any dependency version.What it does:
Light checks on every push and in the merge queue:
lint(clang-format 15 on the changed lines),docs(Doxygen), andworkflows(actionlint, shellcheck, the CI scripts' tests).CPU tests (Linux GCC and Clang, macOS arm64, Windows MSVC), in the merge queue and on every push to
master, and on a pull request when a maintainer comments/cpu-tests.docs/, the templates) passes them in seconds and says why.GPU tests, CUDA and OpenCL on GitHub's T4 runner, when a maintainer comments
/gpu-tests.gpu-testsenvironment.Compile-only builds:
For CUDA they also check the standalone packaging (which cuFFT, cuSOLVER and other libraries get bundled, under which names) and ArrayFire's default architecture list. They run on
masterand with/cpu-tests all. Off until enabled.A nightly: Intel's compiler with MKL, the declared minimum CMake 3.16.3, and an early warning on the newest Ubuntu.
The Matrix Market (sparse) tests run again. SuiteSparse refuses CI's downloads (HTTP 403), so the 9 matrices come from a bundle checked against a SHA-256 manifest, from ci: restore build checks and CPU summation accuracy #3735.
The release workflow still builds the source tarball on tags. It now does so in a read-only job with a script, and uploads with
ghin a separate job.Why:
masterfor months.windows-latestchanges, the CDash server, the SuiteSparse download.ci.sh --docker <job>.Impact on hardware, software and backends:
New functions: none.
Backporting: not needed; this changes CI only.
Related:
lapack-referenceoverlay and triplet, and its Matrix Market bundle. Its library and test fixes stay with that PR, and the formatting baseline stays with style: apply the existing clang-format 15 policy #3736.Not in this PR (possible follow-ups):
Changes to Users
CMakePresets.jsonmoves from format 2 to 6, so using the presets needs CMake 3.25 or newer. The project's own minimum stays 3.16.3, now tested nightly.msvc2019-*is renamedmsvc2022-*; no GitHub runner has Visual Studio 2019 anymore.FFTW/LAPCK/BLASis fixed toFFTW/LAPACK/BLASin three presets. The typo made them skip their dependency checks.ci-*presets, one per CI job. Theninja-*presets are unchanged..github/scripts/ci.sh --docker <job>(--listshows them).CONTRIBUTING.md.master.Checklist
ab4a5063c)CONTRIBUTING.mdTesting evidence
All of this ran in a test copy of the repository with these workflows, and locally with
ci.sh --docker.master:lint,docsandworkflowspassed;windows-cuda (12.9)passed;test_matrixmarket_cpu);linux-cudapassed for 12.9 onmaster, and for 12.9 and 13.1 on cuda: build with CUDA 13 and target Blackwell #3715's code. That includes the Blackwell targets (sm_100, sm_120) and the packaging check;windows-cuda (12.9)built and bundled the right DLLs.master. build: add missing direct includes #3705 fixes that, and the job is allowed to fail.master" line all work. A replaced request is reported as cancelled, at no cost.crtandnvvmcomponents that CUDA 13 split out of nvcc.Findings for the library, shared separately:
thrust/system/cuda/detail/par.h, which CCCL 3.4 removed.compute_53, which nvcc rejects.Jobs and when they run
checks.ymllint,docs,workflowsmaster; the merge queuecpu.ymllinux-gcc-openblas,linux-clang-openblas,macos-arm64-clang-accelerate,windows-msvc-openblas(and achangesjob that decides whether they build)master;/cpu-testscpu.ymlaccelerators (linux-cuda, <version>),accelerators (linux-opencl),accelerators (linux-oneapi),windows-cuda (<version>)masterand/cpu-tests all, only withAF_CI_ACCELERATOR_BUILDS=truecpu-placeholders.ymlcpu-nightly.ymllinux-icx-mkl,linux-gcc-openblas-floor,linux-gcc-openblas-nextarrayfire/arrayfireonlygpu.ymlbuild (…),approve,test (…),summary/gpu-tests, only withAF_CI_GPU_TESTS=truetest-requests.ymldecide,reply,accepted,report/cpu-testsor/gpu-testscommentrelease_src_artifact.ymlUpload release source tarballtest-requests.ymlruns frommasterand builds the pinned commit withmaster's scripts.master's own builds save caches.Requesting tests on a pull request
/cpu-tests/cpu-tests all/gpu-tests/gpu-tests cuda,opencl,all,cuda=13.1,all cuda=13.1cuda=X.Ypicks one CUDA versionmaster.cpu-testsorgpu-testsstatus and a comment.Settings to enable after merging, and how to switch each part off
masterlint,docs,workflows,linux-gcc-openblas,linux-clang-openblas,macos-arm64-clang-accelerate,windows-msvc-openblas; merge queueAF_CI_ACCELERATOR_BUILDStrueAF_CI_CUDA_VERSIONS12.9(add13.1oncemasterpasses the CUDA 13 checks)AF_CI_GPU_RUNNERAF_CI_GPU_TESTStrue, set lastgpu-testsmastertest-requests.ymlandgpu.ymlNON_CODEin.github/scripts/code-changes.py.Commits
Grouped by area; each passes
ci.sh --docker workflowson its own.