Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Shell scripts keep LF endings so that they also run in Git Bash on Windows.
*.sh text eol=lf
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
27 changes: 27 additions & 0 deletions .github/fixtures/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Matrix Market test data

The Matrix Market tests read matrices that `test/CMakeModules/download_sparse_datasets.cmake`
downloads from the [SuiteSparse Matrix Collection](https://sparse.tamu.edu/), whose storage
refuses every download. ArrayFire's [arrayfire-data](https://github.com/arrayfire/arrayfire-data)
repository holds the same files, unchanged, under `matrixmarket/`, with their license
(CC BY 4.0) and citations.

`matrixmarket.json` pins the arrayfire-data commit that CI downloads them from, and records
each matrix's size and SHA-256, and its original SuiteSparse URL and archive SHA-256. Before
every build, `ci.sh` runs `.github/scripts/check-matrix-data.py`:

- **While `commit` is empty,** it prints that no commit is pinned, and the Matrix Market
tests are off (`AF_TEST_WITH_MTX_FILES=OFF`).
- **With a commit,** it downloads each matrix from
`https://raw.githubusercontent.com/arrayfire/arrayfire-data/<commit>/matrixmarket/`, checks
it against the manifest, and restores it into the build tree. The tests are then on, and the
CI configurations point their download at the restored files
(`.github/scripts/matrixmarket.cmake` and the `ci` preset), so the configure downloads
nothing. A failed download or check fails the build, naming the file.

To turn the tests on, set `commit` to a full commit SHA of arrayfire-data that has
`matrixmarket/`. When the tests' list of matrices changes, update the manifest,
`matrixmarket.cmake` and the `ci` preset together; `check-matrix-data-test.py` checks that
they agree.

The manifest comes from ArrayFire's fix/restore-ci branch.
90 changes: 90 additions & 0 deletions .github/fixtures/matrixmarket.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AF_TEST_WITH_MTX_FILES=OFF

"source": {
"repository": "arrayfire/arrayfire-data",
"path": "matrixmarket/",
"commit": "2723164f7a8b8d2ebf0f50c941cbce1f71809673"
},
"files": {
"JGD_Kocay/Trec4/Trec4.mtx": {
"sha256": "c33e82bc1728021c4a29937f502a610dda9c7a81748bff47badc65e4bf3b7408",
"bytes": 2263,
"rows": 2,
"columns": 3,
"entries": 3,
"download_url": "https://sparse.tamu.edu/MM/JGD_Kocay/Trec4.tar.gz",
"original_archive_sha256": "f49eedd1c2904df742cd492e0e30fc928bcf904f0ec1c4225a05c9e8d8f1e30d"
},
"HB/bcsstm02/bcsstm02.mtx": {
"sha256": "c8c629d8ea30269b3ee71a0fce17914e463359e817ae4553d3745b98f0cddae1",
"bytes": 1884,
"rows": 66,
"columns": 66,
"entries": 66,
"download_url": "https://sparse.tamu.edu/MM/HB/bcsstm02.tar.gz",
"original_archive_sha256": "72897c2859efed23dad58748683dc094fbbf2ea63532f83b8deedbcb961c7c98"
},
"HB/young4c/young4c.mtx": {
"sha256": "06db1d6ff551065237809e3d79991e1788ab24eae295a1cd3eb37dab342a59e5",
"bytes": 61922,
"rows": 841,
"columns": 841,
"entries": 4089,
"download_url": "https://sparse.tamu.edu/MM/HB/young4c.tar.gz",
"original_archive_sha256": "1f53a17ddfc4fdfd291db74ef02eb8c3edfc65b799e62a9feb3794fe42e05c3e"
},
"LPnetlib/lpi_vol1/lpi_vol1.mtx": {
"sha256": "325ad6f923f11926add3481621f90eb08003377f69377756477fc05067bac3e0",
"bytes": 23421,
"rows": 323,
"columns": 464,
"entries": 1646,
"download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_vol1.tar.gz",
"original_archive_sha256": "0e9a9897560ad0829030dad1adea06cd4fb0dd50fcc3910132d9bc00f1db6bc9"
},
"LPnetlib/lpi_qual/lpi_qual.mtx": {
"sha256": "195e2e19664ebf88fb0eeb25e882934fb05e176d7c0b867752a7610caa57dbde",
"bytes": 23342,
"rows": 323,
"columns": 464,
"entries": 1646,
"download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_qual.tar.gz",
"original_archive_sha256": "54fb7b137891c1d82fe349854d0a7719cfe1e8dab1e23ab3e4ec230a29341873"
},
"Sandia/oscil_dcop_12/oscil_dcop_12.mtx": {
"sha256": "8da3dcf47cc90668c9d808c389b261e0ef32cb6411095810f96a71cde9161155",
"bytes": 31102,
"rows": 430,
"columns": 430,
"entries": 1544,
"download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_12.tar.gz",
"original_archive_sha256": "fad82385c1d134640e581f698eac6e835a2d9d9a9b7d6db9892d4b9919630da7"
},
"Sandia/oscil_dcop_42/oscil_dcop_42.mtx": {
"sha256": "a1cfe04f2891763afea28064ca8c1f3337e86aac652fdcdf86afe81577966cbf",
"bytes": 31086,
"rows": 430,
"columns": 430,
"entries": 1544,
"download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_42.tar.gz",
"original_archive_sha256": "3a7fce53d2bfbe26dd0bde2995f7b52078ac33359ced79680581ce90fced0b17"
},
"QCD/conf6_0-4x4-20/conf6_0-4x4-20.mtx": {
"sha256": "cffe2924b9f21f869b2ee8fb8d00b042653ac1a62775d5ac40d0b6d1c527a152",
"bytes": 5087623,
"rows": 3072,
"columns": 3072,
"entries": 119808,
"download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-20.tar.gz",
"original_archive_sha256": "f6be76ab90502bbd6d645d6cc925ff01efac8d5437927b9e0ebfb57f07b7a4bc"
},
"QCD/conf6_0-4x4-30/conf6_0-4x4-30.mtx": {
"sha256": "5ede088c30cd221bad591af166b76f38e5005e86d9a6f610b3b046815fc95738",
"bytes": 5087643,
"rows": 3072,
"columns": 3072,
"entries": 119808,
"download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-30.tar.gz",
"original_archive_sha256": "6c8576ee077339c96747b861a298dc774a1dff7aaf9a1068b91932f89e48ff96"
}
}
}
47 changes: 47 additions & 0 deletions .github/scripts/check-bash32.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Checks that the scripts the macOS job runs work with macOS's bash 3.2: it runs
# itself in the official bash:3.2 image and checks their syntax, ci.sh --list,
# checkout-commit.sh's handling of a short SHA, and that they use no syntax added
# in bash 4. It must itself run on bash 3.2.
#
# The image is pinned by digest. To update it: docker pull bash:3.2, then
# docker inspect --format '{{index .RepoDigests 0}}' bash:3.2.
#
# Usage: check-bash32.sh [<repository>] (default: this one)
set -euo pipefail

image=bash:3.2@sha256:b6252c62fc86114f93f02104a486c71e6569371d95950ea7479a2ffd6b47f16a
scripts="checkout-commit.sh ci.sh install-deps-macos.sh"

if [[ ${1:-} != --inside ]]; then
root=$(cd "${1:-$(dirname "$0")/../..}" && pwd)
exec docker run --rm --network none --memory=256m --memory-swap=256m \
-v "$root:/ci:ro" "$image" bash /ci/.github/scripts/check-bash32.sh --inside
fi

cd /ci/.github/scripts
echo "bash $BASH_VERSION"
failed=0
fail() { echo "FAIL: $*"; failed=1; }

for script in $scripts; do
bash -n "$script" || fail "bash -n $script"
done
./ci.sh --list > /dev/null || fail "ci.sh --list"
out=$(./checkout-commit.sh 0123ABC /tmp/checkout 2>&1) || true
[[ $out == "not a full 40-character commit SHA: 0123ABC" ]] ||
fail "checkout-commit.sh with a short SHA printed: $out"

# Case conversion, associative arrays and namerefs, mapfile, -v tests, @
# transformations, &>> and |&, negative indices, wait -n, coproc, ;& and ;;&.
bash4='\$\{[A-Za-z_0-9]+(,,?|\^\^?)\}|\$\{[A-Za-z_0-9]+@[QEPAKa]\}|(declare|local|typeset) -[a-zA-Z]*[Anlu]|mapfile|readarray|\[\[ -v |test -v |&>>|\|&|\[-[0-9]+\]|wait -n|coproc|;;?&'
# shellcheck disable=SC2086
if grep -q -E "$bash4" $scripts; then
fail "syntax that needs bash 4:"
grep -n -E "$bash4" $scripts | sed 's/^/ /'
fi

if ((failed)); then
exit 1
fi
echo "The macOS job's scripts ($scripts) pass under bash 3.2."
86 changes: 86 additions & 0 deletions .github/scripts/check-cuda-release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
# Configures the CUDA backend the way users and the release installers get it,
# with ArrayFire's default architecture list and AF_INSTALL_STANDALONE=ON, and
# checks two things that the CI builds' explicit architecture list and plain
# install would hide:
#
# - The default architecture list: prints the flags CMake chose and compiles one
# CUDA object of afcuda with them, which fails when the list holds an
# architecture the toolkit dropped. Without a GPU, the default ("Auto") is the
# common list for the toolkit.
# - The bundled CUDA libraries: installs the cuda_dependencies component into a
# temporary prefix and checks that every library afcu_collect_libs bundles on
# Linux (cuFFT, cuBLAS, cuBLASLt, cuSOLVER, cuSPARSE, nvJitLink, NVRTC and its
# builtins) is there once, under the toolkit's soname, which is the name
# libafcuda loads it by.
#
# Only a configure and one compile: it needs the CUDA toolkit, not a build or a GPU.
#
# Usage: check-cuda-release.sh <source dir> <build dir>
set -euo pipefail

[[ $# -eq 2 ]] || { echo "usage: $0 <source dir> <build dir>" >&2; exit 2; }
source=$1 build=$2
cuda=/usr/local/cuda
failed=

cmake -S "$source" -B "$build" -G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CUDA_COMPILER="$cuda/bin/nvcc" \
-DAF_BUILD_CPU=OFF -DAF_BUILD_CUDA=ON -DAF_BUILD_OPENCL=OFF -DAF_BUILD_ONEAPI=OFF \
-DAF_BUILD_UNIFIED=OFF -DAF_BUILD_FORGE=OFF -DAF_BUILD_EXAMPLES=OFF -DAF_BUILD_DOCS=OFF \
-DBUILD_TESTING=OFF -DAF_WITH_CUDNN=OFF -DAF_INSTALL_STANDALONE=ON > "$build.configure.log" 2>&1 ||
{ tail -n 30 "$build.configure.log"; echo "The configure failed; see $build.configure.log" >&2; exit 1; }
grep -m 1 -E 'GPU detection|common architectures' "$build.configure.log" || true

echo "=== The default architecture list"
object=src/backend/cuda/CMakeFiles/afcuda.dir/cufft.cu.o
flags=$(grep -A 8 "^build $object:" "$build/build.ninja" | grep -m 1 '^ *FLAGS = ' |
grep -o -- '--generate-code=arch=[^ "]*\|-gencode[= ]arch=[^ "]*' | sed 's/^--generate-code=//' || true)
echo "CUDA_architecture_build_targets=$(sed -n 's/^CUDA_architecture_build_targets:[A-Z]*=//p' "$build/CMakeCache.txt")"
echo "Flags for $object:"
while IFS= read -r flag; do echo " $flag"; done <<< "${flags:-(none found)}"
if ninja -C "$build" "$object" > "$build.compile.log" 2>&1; then
echo "Compiling $object with them: passed"
else
grep -E 'nvcc fatal|error' "$build.compile.log" | head -n 5
echo "Compiling $object with them: FAILED" >&2
failed=1
fi

echo "=== The bundled CUDA libraries (AF_INSTALL_STANDALONE=ON, component cuda_dependencies)"
prefix=$(mktemp -d)
if ! cmake --install "$build" --component cuda_dependencies --prefix "$prefix" > "$build.install.log" 2>&1; then
grep -E 'CMake Error|cannot find|not found' "$build.install.log" | head -n 5
echo "Installing cuda_dependencies: FAILED" >&2
failed=1
fi
soname() { { readelf -d "$1" 2> /dev/null || true; } | sed -n 's/.*(SONAME).*\[\(.*\)\]/\1/p'; }
printf '%-15s %-28s %-28s %s\n' library "toolkit soname" bundled "bundled file's soname"
for lib in cufft cublas cublasLt cusolver cusparse nvJitLink nvrtc nvrtc-builtins; do
expected=$(soname "$(readlink -f "$cuda/lib64/lib$lib.so")")
expected=${expected:-"(no lib$lib.so)"}
bundled=$(cd "$prefix" && find . -name "lib$lib.so*" | sed 's|^\./||' | sort | tr '\n' ' ')
bundled=${bundled% }
if [[ -z $bundled ]]; then
printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "MISSING" "-"
failed=1
elif [[ $bundled == *' '* ]]; then
printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "$bundled" "(more than one)"
failed=1
else
actual=$(soname "$prefix/$bundled")
status=
[[ ${bundled##*/} == "$expected" && $actual == "$expected" ]] || { status=" <- WRONG NAME"; failed=1; }
printf '%-15s %-28s %-28s %s%s\n' "$lib" "$expected" "$bundled" "$actual" "$status"
fi
done
others=$(cd "$prefix" && find . -type f | sed 's|^\./||' |
grep -v -E '/lib(cufft|cublas|cublasLt|cusolver|cusparse|nvJitLink|nvrtc|nvrtc-builtins)\.so' || true)
[[ -z $others ]] || echo "Also bundled: $others"
rm -rf "$prefix"

if [[ -n $failed ]]; then
echo "The release configuration check failed." >&2
exit 1
fi
echo "The release configuration check passed."
Loading