Repository navigation
Rework the CI: light checks on every push, CPU tests in the merge queue, GPU tests on request #3738
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
f1a350c
Replace the old CI workflows with ci.sh and its scripts
roaffix 6e12a47
Fix the presets' LAPACK value, the clFFT pin, Doxygen and .gitignore
roaffix 3398a6f
Add CMake presets for the CI configurations
roaffix bc5b8ee
Add the light checks: lint, docs and workflows
roaffix 8884070
Take the LAPACK overlay and triplet from the fix/restore-ci branch
roaffix 8bb1330
Get the Matrix Market test data from a pinned arrayfire-data commit
roaffix cdd9435
Add the CUDA release checks and the Windows CUDA scripts
roaffix 016a5c7
Add a GPU test workflow with one approval per request
roaffix e8ca974
Run the CPU tests in the merge queue, on master and nightly
roaffix a36915c
Run CPU and GPU tests on request with /cpu-tests and /gpu-tests
roaffix 9fdfc39
Describe CI in CONTRIBUTING.md
roaffix 847d393
Let Dependabot keep the pinned actions current
roaffix 6c1328f
allow /cpu-tests and /gpu-tests to specify SHA
syurkevi d566d0d
build windows CI vcpkg dependencies for release only
syurkevi da4c616
keep CMakePresets.json at format 2
syurkevi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Shell scripts keep LF endings so that they also run in Git Bash on Windows. | ||
| *.sh text eol=lf |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| version: 2 | ||
| updates: | ||
| - package-ecosystem: github-actions | ||
| directory: / | ||
| schedule: | ||
| interval: weekly |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Matrix Market test data | ||
|
|
||
| The Matrix Market tests read matrices that `test/CMakeModules/download_sparse_datasets.cmake` | ||
| downloads from the [SuiteSparse Matrix Collection](https://sparse.tamu.edu/), whose storage | ||
| refuses every download. ArrayFire's [arrayfire-data](https://github.com/arrayfire/arrayfire-data) | ||
| repository holds the same files, unchanged, under `matrixmarket/`, with their license | ||
| (CC BY 4.0) and citations. | ||
|
|
||
| `matrixmarket.json` pins the arrayfire-data commit that CI downloads them from, and records | ||
| each matrix's size and SHA-256, and its original SuiteSparse URL and archive SHA-256. Before | ||
| every build, `ci.sh` runs `.github/scripts/check-matrix-data.py`: | ||
|
|
||
| - **While `commit` is empty,** it prints that no commit is pinned, and the Matrix Market | ||
| tests are off (`AF_TEST_WITH_MTX_FILES=OFF`). | ||
| - **With a commit,** it downloads each matrix from | ||
| `https://raw.githubusercontent.com/arrayfire/arrayfire-data/<commit>/matrixmarket/`, checks | ||
| it against the manifest, and restores it into the build tree. The tests are then on, and the | ||
| CI configurations point their download at the restored files | ||
| (`.github/scripts/matrixmarket.cmake` and the `ci` preset), so the configure downloads | ||
| nothing. A failed download or check fails the build, naming the file. | ||
|
|
||
| To turn the tests on, set `commit` to a full commit SHA of arrayfire-data that has | ||
| `matrixmarket/`. When the tests' list of matrices changes, update the manifest, | ||
| `matrixmarket.cmake` and the `ci` preset together; `check-matrix-data-test.py` checks that | ||
| they agree. | ||
|
|
||
| The manifest comes from ArrayFire's fix/restore-ci branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| { | ||
| "source": { | ||
| "repository": "arrayfire/arrayfire-data", | ||
| "path": "matrixmarket/", | ||
| "commit": "2723164f7a8b8d2ebf0f50c941cbce1f71809673" | ||
| }, | ||
| "files": { | ||
| "JGD_Kocay/Trec4/Trec4.mtx": { | ||
| "sha256": "c33e82bc1728021c4a29937f502a610dda9c7a81748bff47badc65e4bf3b7408", | ||
| "bytes": 2263, | ||
| "rows": 2, | ||
| "columns": 3, | ||
| "entries": 3, | ||
| "download_url": "https://sparse.tamu.edu/MM/JGD_Kocay/Trec4.tar.gz", | ||
| "original_archive_sha256": "f49eedd1c2904df742cd492e0e30fc928bcf904f0ec1c4225a05c9e8d8f1e30d" | ||
| }, | ||
| "HB/bcsstm02/bcsstm02.mtx": { | ||
| "sha256": "c8c629d8ea30269b3ee71a0fce17914e463359e817ae4553d3745b98f0cddae1", | ||
| "bytes": 1884, | ||
| "rows": 66, | ||
| "columns": 66, | ||
| "entries": 66, | ||
| "download_url": "https://sparse.tamu.edu/MM/HB/bcsstm02.tar.gz", | ||
| "original_archive_sha256": "72897c2859efed23dad58748683dc094fbbf2ea63532f83b8deedbcb961c7c98" | ||
| }, | ||
| "HB/young4c/young4c.mtx": { | ||
| "sha256": "06db1d6ff551065237809e3d79991e1788ab24eae295a1cd3eb37dab342a59e5", | ||
| "bytes": 61922, | ||
| "rows": 841, | ||
| "columns": 841, | ||
| "entries": 4089, | ||
| "download_url": "https://sparse.tamu.edu/MM/HB/young4c.tar.gz", | ||
| "original_archive_sha256": "1f53a17ddfc4fdfd291db74ef02eb8c3edfc65b799e62a9feb3794fe42e05c3e" | ||
| }, | ||
| "LPnetlib/lpi_vol1/lpi_vol1.mtx": { | ||
| "sha256": "325ad6f923f11926add3481621f90eb08003377f69377756477fc05067bac3e0", | ||
| "bytes": 23421, | ||
| "rows": 323, | ||
| "columns": 464, | ||
| "entries": 1646, | ||
| "download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_vol1.tar.gz", | ||
| "original_archive_sha256": "0e9a9897560ad0829030dad1adea06cd4fb0dd50fcc3910132d9bc00f1db6bc9" | ||
| }, | ||
| "LPnetlib/lpi_qual/lpi_qual.mtx": { | ||
| "sha256": "195e2e19664ebf88fb0eeb25e882934fb05e176d7c0b867752a7610caa57dbde", | ||
| "bytes": 23342, | ||
| "rows": 323, | ||
| "columns": 464, | ||
| "entries": 1646, | ||
| "download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_qual.tar.gz", | ||
| "original_archive_sha256": "54fb7b137891c1d82fe349854d0a7719cfe1e8dab1e23ab3e4ec230a29341873" | ||
| }, | ||
| "Sandia/oscil_dcop_12/oscil_dcop_12.mtx": { | ||
| "sha256": "8da3dcf47cc90668c9d808c389b261e0ef32cb6411095810f96a71cde9161155", | ||
| "bytes": 31102, | ||
| "rows": 430, | ||
| "columns": 430, | ||
| "entries": 1544, | ||
| "download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_12.tar.gz", | ||
| "original_archive_sha256": "fad82385c1d134640e581f698eac6e835a2d9d9a9b7d6db9892d4b9919630da7" | ||
| }, | ||
| "Sandia/oscil_dcop_42/oscil_dcop_42.mtx": { | ||
| "sha256": "a1cfe04f2891763afea28064ca8c1f3337e86aac652fdcdf86afe81577966cbf", | ||
| "bytes": 31086, | ||
| "rows": 430, | ||
| "columns": 430, | ||
| "entries": 1544, | ||
| "download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_42.tar.gz", | ||
| "original_archive_sha256": "3a7fce53d2bfbe26dd0bde2995f7b52078ac33359ced79680581ce90fced0b17" | ||
| }, | ||
| "QCD/conf6_0-4x4-20/conf6_0-4x4-20.mtx": { | ||
| "sha256": "cffe2924b9f21f869b2ee8fb8d00b042653ac1a62775d5ac40d0b6d1c527a152", | ||
| "bytes": 5087623, | ||
| "rows": 3072, | ||
| "columns": 3072, | ||
| "entries": 119808, | ||
| "download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-20.tar.gz", | ||
| "original_archive_sha256": "f6be76ab90502bbd6d645d6cc925ff01efac8d5437927b9e0ebfb57f07b7a4bc" | ||
| }, | ||
| "QCD/conf6_0-4x4-30/conf6_0-4x4-30.mtx": { | ||
| "sha256": "5ede088c30cd221bad591af166b76f38e5005e86d9a6f610b3b046815fc95738", | ||
| "bytes": 5087643, | ||
| "rows": 3072, | ||
| "columns": 3072, | ||
| "entries": 119808, | ||
| "download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-30.tar.gz", | ||
| "original_archive_sha256": "6c8576ee077339c96747b861a298dc774a1dff7aaf9a1068b91932f89e48ff96" | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #!/usr/bin/env bash | ||
| # Checks that the scripts the macOS job runs work with macOS's bash 3.2: it runs | ||
| # itself in the official bash:3.2 image and checks their syntax, ci.sh --list, | ||
| # checkout-commit.sh's handling of a short SHA, and that they use no syntax added | ||
| # in bash 4. It must itself run on bash 3.2. | ||
| # | ||
| # The image is pinned by digest. To update it: docker pull bash:3.2, then | ||
| # docker inspect --format '{{index .RepoDigests 0}}' bash:3.2. | ||
| # | ||
| # Usage: check-bash32.sh [<repository>] (default: this one) | ||
| set -euo pipefail | ||
|
|
||
| image=bash:3.2@sha256:b6252c62fc86114f93f02104a486c71e6569371d95950ea7479a2ffd6b47f16a | ||
| scripts="checkout-commit.sh ci.sh install-deps-macos.sh" | ||
|
|
||
| if [[ ${1:-} != --inside ]]; then | ||
| root=$(cd "${1:-$(dirname "$0")/../..}" && pwd) | ||
| exec docker run --rm --network none --memory=256m --memory-swap=256m \ | ||
| -v "$root:/ci:ro" "$image" bash /ci/.github/scripts/check-bash32.sh --inside | ||
| fi | ||
|
|
||
| cd /ci/.github/scripts | ||
| echo "bash $BASH_VERSION" | ||
| failed=0 | ||
| fail() { echo "FAIL: $*"; failed=1; } | ||
|
|
||
| for script in $scripts; do | ||
| bash -n "$script" || fail "bash -n $script" | ||
| done | ||
| ./ci.sh --list > /dev/null || fail "ci.sh --list" | ||
| out=$(./checkout-commit.sh 0123ABC /tmp/checkout 2>&1) || true | ||
| [[ $out == "not a full 40-character commit SHA: 0123ABC" ]] || | ||
| fail "checkout-commit.sh with a short SHA printed: $out" | ||
|
|
||
| # Case conversion, associative arrays and namerefs, mapfile, -v tests, @ | ||
| # transformations, &>> and |&, negative indices, wait -n, coproc, ;& and ;;&. | ||
| bash4='\$\{[A-Za-z_0-9]+(,,?|\^\^?)\}|\$\{[A-Za-z_0-9]+@[QEPAKa]\}|(declare|local|typeset) -[a-zA-Z]*[Anlu]|mapfile|readarray|\[\[ -v |test -v |&>>|\|&|\[-[0-9]+\]|wait -n|coproc|;;?&' | ||
| # shellcheck disable=SC2086 | ||
| if grep -q -E "$bash4" $scripts; then | ||
| fail "syntax that needs bash 4:" | ||
| grep -n -E "$bash4" $scripts | sed 's/^/ /' | ||
| fi | ||
|
|
||
| if ((failed)); then | ||
| exit 1 | ||
| fi | ||
| echo "The macOS job's scripts ($scripts) pass under bash 3.2." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| #!/usr/bin/env bash | ||
| # Configures the CUDA backend the way users and the release installers get it, | ||
| # with ArrayFire's default architecture list and AF_INSTALL_STANDALONE=ON, and | ||
| # checks two things that the CI builds' explicit architecture list and plain | ||
| # install would hide: | ||
| # | ||
| # - The default architecture list: prints the flags CMake chose and compiles one | ||
| # CUDA object of afcuda with them, which fails when the list holds an | ||
| # architecture the toolkit dropped. Without a GPU, the default ("Auto") is the | ||
| # common list for the toolkit. | ||
| # - The bundled CUDA libraries: installs the cuda_dependencies component into a | ||
| # temporary prefix and checks that every library afcu_collect_libs bundles on | ||
| # Linux (cuFFT, cuBLAS, cuBLASLt, cuSOLVER, cuSPARSE, nvJitLink, NVRTC and its | ||
| # builtins) is there once, under the toolkit's soname, which is the name | ||
| # libafcuda loads it by. | ||
| # | ||
| # Only a configure and one compile: it needs the CUDA toolkit, not a build or a GPU. | ||
| # | ||
| # Usage: check-cuda-release.sh <source dir> <build dir> | ||
| set -euo pipefail | ||
|
|
||
| [[ $# -eq 2 ]] || { echo "usage: $0 <source dir> <build dir>" >&2; exit 2; } | ||
| source=$1 build=$2 | ||
| cuda=/usr/local/cuda | ||
| failed= | ||
|
|
||
| cmake -S "$source" -B "$build" -G Ninja -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CUDA_COMPILER="$cuda/bin/nvcc" \ | ||
| -DAF_BUILD_CPU=OFF -DAF_BUILD_CUDA=ON -DAF_BUILD_OPENCL=OFF -DAF_BUILD_ONEAPI=OFF \ | ||
| -DAF_BUILD_UNIFIED=OFF -DAF_BUILD_FORGE=OFF -DAF_BUILD_EXAMPLES=OFF -DAF_BUILD_DOCS=OFF \ | ||
| -DBUILD_TESTING=OFF -DAF_WITH_CUDNN=OFF -DAF_INSTALL_STANDALONE=ON > "$build.configure.log" 2>&1 || | ||
| { tail -n 30 "$build.configure.log"; echo "The configure failed; see $build.configure.log" >&2; exit 1; } | ||
| grep -m 1 -E 'GPU detection|common architectures' "$build.configure.log" || true | ||
|
|
||
| echo "=== The default architecture list" | ||
| object=src/backend/cuda/CMakeFiles/afcuda.dir/cufft.cu.o | ||
| flags=$(grep -A 8 "^build $object:" "$build/build.ninja" | grep -m 1 '^ *FLAGS = ' | | ||
| grep -o -- '--generate-code=arch=[^ "]*\|-gencode[= ]arch=[^ "]*' | sed 's/^--generate-code=//' || true) | ||
| echo "CUDA_architecture_build_targets=$(sed -n 's/^CUDA_architecture_build_targets:[A-Z]*=//p' "$build/CMakeCache.txt")" | ||
| echo "Flags for $object:" | ||
| while IFS= read -r flag; do echo " $flag"; done <<< "${flags:-(none found)}" | ||
| if ninja -C "$build" "$object" > "$build.compile.log" 2>&1; then | ||
| echo "Compiling $object with them: passed" | ||
| else | ||
| grep -E 'nvcc fatal|error' "$build.compile.log" | head -n 5 | ||
| echo "Compiling $object with them: FAILED" >&2 | ||
| failed=1 | ||
| fi | ||
|
|
||
| echo "=== The bundled CUDA libraries (AF_INSTALL_STANDALONE=ON, component cuda_dependencies)" | ||
| prefix=$(mktemp -d) | ||
| if ! cmake --install "$build" --component cuda_dependencies --prefix "$prefix" > "$build.install.log" 2>&1; then | ||
| grep -E 'CMake Error|cannot find|not found' "$build.install.log" | head -n 5 | ||
| echo "Installing cuda_dependencies: FAILED" >&2 | ||
| failed=1 | ||
| fi | ||
| soname() { { readelf -d "$1" 2> /dev/null || true; } | sed -n 's/.*(SONAME).*\[\(.*\)\]/\1/p'; } | ||
| printf '%-15s %-28s %-28s %s\n' library "toolkit soname" bundled "bundled file's soname" | ||
| for lib in cufft cublas cublasLt cusolver cusparse nvJitLink nvrtc nvrtc-builtins; do | ||
| expected=$(soname "$(readlink -f "$cuda/lib64/lib$lib.so")") | ||
| expected=${expected:-"(no lib$lib.so)"} | ||
| bundled=$(cd "$prefix" && find . -name "lib$lib.so*" | sed 's|^\./||' | sort | tr '\n' ' ') | ||
| bundled=${bundled% } | ||
| if [[ -z $bundled ]]; then | ||
| printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "MISSING" "-" | ||
| failed=1 | ||
| elif [[ $bundled == *' '* ]]; then | ||
| printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "$bundled" "(more than one)" | ||
| failed=1 | ||
| else | ||
| actual=$(soname "$prefix/$bundled") | ||
| status= | ||
| [[ ${bundled##*/} == "$expected" && $actual == "$expected" ]] || { status=" <- WRONG NAME"; failed=1; } | ||
| printf '%-15s %-28s %-28s %s%s\n' "$lib" "$expected" "$bundled" "$actual" "$status" | ||
| fi | ||
| done | ||
| others=$(cd "$prefix" && find . -type f | sed 's|^\./||' | | ||
| grep -v -E '/lib(cufft|cublas|cublasLt|cusolver|cusparse|nvJitLink|nvrtc|nvrtc-builtins)\.so' || true) | ||
| [[ -z $others ]] || echo "Also bundled: $others" | ||
| rm -rf "$prefix" | ||
|
|
||
| if [[ -n $failed ]]; then | ||
| echo "The release configuration check failed." >&2 | ||
| exit 1 | ||
| fi | ||
| echo "The release configuration check passed." |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AF_TEST_WITH_MTX_FILES=OFF