Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Shell scripts keep LF endings so that they also run in Git Bash on Windows.
*.sh text eol=lf
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
21 changes: 21 additions & 0 deletions .github/fixtures/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Matrix Market test data

`matrixmarket.tar.gz` holds the matrices that the tests download in
`test/CMakeModules/download_sparse_datasets.cmake`. They are public scientific
datasets from the [SuiteSparse Matrix Collection](https://sparse.tamu.edu/),
which refuses downloads from CI. Each file keeps its original header, author
credits and collection metadata.

`matrixmarket.json` records each matrix's source URL, the SHA-256 of the file and
of its original archive, its size, dimensions and entry count, and the SHA-256 of
the bundle. Before every build, `ci.sh` runs `.github/scripts/check-matrix-data.py`,
which checks the bundle and every matrix against it, fails naming any file that
differs, and restores the matrices into the build tree. The CI configurations point
the tests' download at them (`.github/scripts/matrixmarket.cmake` and the `ci`
preset), so the configure downloads nothing.

When the tests' list of matrices changes, update the bundle, the manifest,
`matrixmarket.cmake` and the `ci` preset together; `check-matrix-data-test.py`
checks that they agree.

The bundle and the manifest come from ArrayFire's fix/restore-ci branch.
87 changes: 87 additions & 0 deletions .github/fixtures/matrixmarket.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
{
"source": "https://sparse.tamu.edu/",
"archive_sha256": "a6806c0e5f47be63f60713686382414f372bc98afe7410cbafe72362fcc22c84",
"files": {
"JGD_Kocay/Trec4/Trec4.mtx": {
"sha256": "c33e82bc1728021c4a29937f502a610dda9c7a81748bff47badc65e4bf3b7408",
"bytes": 2263,
"rows": 2,
"columns": 3,
"entries": 3,
"download_url": "https://sparse.tamu.edu/MM/JGD_Kocay/Trec4.tar.gz",
"original_archive_sha256": "f49eedd1c2904df742cd492e0e30fc928bcf904f0ec1c4225a05c9e8d8f1e30d"
},
"HB/bcsstm02/bcsstm02.mtx": {
"sha256": "c8c629d8ea30269b3ee71a0fce17914e463359e817ae4553d3745b98f0cddae1",
"bytes": 1884,
"rows": 66,
"columns": 66,
"entries": 66,
"download_url": "https://sparse.tamu.edu/MM/HB/bcsstm02.tar.gz",
"original_archive_sha256": "72897c2859efed23dad58748683dc094fbbf2ea63532f83b8deedbcb961c7c98"
},
"HB/young4c/young4c.mtx": {
"sha256": "06db1d6ff551065237809e3d79991e1788ab24eae295a1cd3eb37dab342a59e5",
"bytes": 61922,
"rows": 841,
"columns": 841,
"entries": 4089,
"download_url": "https://sparse.tamu.edu/MM/HB/young4c.tar.gz",
"original_archive_sha256": "1f53a17ddfc4fdfd291db74ef02eb8c3edfc65b799e62a9feb3794fe42e05c3e"
},
"LPnetlib/lpi_vol1/lpi_vol1.mtx": {
"sha256": "325ad6f923f11926add3481621f90eb08003377f69377756477fc05067bac3e0",
"bytes": 23421,
"rows": 323,
"columns": 464,
"entries": 1646,
"download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_vol1.tar.gz",
"original_archive_sha256": "0e9a9897560ad0829030dad1adea06cd4fb0dd50fcc3910132d9bc00f1db6bc9"
},
"LPnetlib/lpi_qual/lpi_qual.mtx": {
"sha256": "195e2e19664ebf88fb0eeb25e882934fb05e176d7c0b867752a7610caa57dbde",
"bytes": 23342,
"rows": 323,
"columns": 464,
"entries": 1646,
"download_url": "https://sparse.tamu.edu/MM/LPnetlib/lpi_qual.tar.gz",
"original_archive_sha256": "54fb7b137891c1d82fe349854d0a7719cfe1e8dab1e23ab3e4ec230a29341873"
},
"Sandia/oscil_dcop_12/oscil_dcop_12.mtx": {
"sha256": "8da3dcf47cc90668c9d808c389b261e0ef32cb6411095810f96a71cde9161155",
"bytes": 31102,
"rows": 430,
"columns": 430,
"entries": 1544,
"download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_12.tar.gz",
"original_archive_sha256": "fad82385c1d134640e581f698eac6e835a2d9d9a9b7d6db9892d4b9919630da7"
},
"Sandia/oscil_dcop_42/oscil_dcop_42.mtx": {
"sha256": "a1cfe04f2891763afea28064ca8c1f3337e86aac652fdcdf86afe81577966cbf",
"bytes": 31086,
"rows": 430,
"columns": 430,
"entries": 1544,
"download_url": "https://sparse.tamu.edu/MM/Sandia/oscil_dcop_42.tar.gz",
"original_archive_sha256": "3a7fce53d2bfbe26dd0bde2995f7b52078ac33359ced79680581ce90fced0b17"
},
"QCD/conf6_0-4x4-20/conf6_0-4x4-20.mtx": {
"sha256": "cffe2924b9f21f869b2ee8fb8d00b042653ac1a62775d5ac40d0b6d1c527a152",
"bytes": 5087623,
"rows": 3072,
"columns": 3072,
"entries": 119808,
"download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-20.tar.gz",
"original_archive_sha256": "f6be76ab90502bbd6d645d6cc925ff01efac8d5437927b9e0ebfb57f07b7a4bc"
},
"QCD/conf6_0-4x4-30/conf6_0-4x4-30.mtx": {
"sha256": "5ede088c30cd221bad591af166b76f38e5005e86d9a6f610b3b046815fc95738",
"bytes": 5087643,
"rows": 3072,
"columns": 3072,
"entries": 119808,
"download_url": "https://sparse.tamu.edu/MM/QCD/conf6_0-4x4-30.tar.gz",
"original_archive_sha256": "6c8576ee077339c96747b861a298dc774a1dff7aaf9a1068b91932f89e48ff96"
}
}
}
Binary file added .github/fixtures/matrixmarket.tar.gz
Binary file not shown.
47 changes: 47 additions & 0 deletions .github/scripts/check-bash32.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Checks that the scripts the macOS job runs work with macOS's bash 3.2: it runs
# itself in the official bash:3.2 image and checks their syntax, ci.sh --list,
# checkout-commit.sh's handling of a short SHA, and that they use no syntax added
# in bash 4. It must itself run on bash 3.2.
#
# The image is pinned by digest. To update it: docker pull bash:3.2, then
# docker inspect --format '{{index .RepoDigests 0}}' bash:3.2.
#
# Usage: check-bash32.sh [<repository>] (default: this one)
set -euo pipefail

image=bash:3.2@sha256:b6252c62fc86114f93f02104a486c71e6569371d95950ea7479a2ffd6b47f16a
scripts="checkout-commit.sh ci.sh install-deps-macos.sh"

if [[ ${1:-} != --inside ]]; then
root=$(cd "${1:-$(dirname "$0")/../..}" && pwd)
exec docker run --rm --network none --memory=256m --memory-swap=256m \
-v "$root:/ci:ro" "$image" bash /ci/.github/scripts/check-bash32.sh --inside
fi

cd /ci/.github/scripts
echo "bash $BASH_VERSION"
failed=0
fail() { echo "FAIL: $*"; failed=1; }

for script in $scripts; do
bash -n "$script" || fail "bash -n $script"
done
./ci.sh --list > /dev/null || fail "ci.sh --list"
out=$(./checkout-commit.sh 0123ABC /tmp/checkout 2>&1) || true
[[ $out == "not a full 40-character commit SHA: 0123ABC" ]] ||
fail "checkout-commit.sh with a short SHA printed: $out"

# Case conversion, associative arrays and namerefs, mapfile, -v tests, @
# transformations, &>> and |&, negative indices, wait -n, coproc, ;& and ;;&.
bash4='\$\{[A-Za-z_0-9]+(,,?|\^\^?)\}|\$\{[A-Za-z_0-9]+@[QEPAKa]\}|(declare|local|typeset) -[a-zA-Z]*[Anlu]|mapfile|readarray|\[\[ -v |test -v |&>>|\|&|\[-[0-9]+\]|wait -n|coproc|;;?&'
# shellcheck disable=SC2086
if grep -q -E "$bash4" $scripts; then
fail "syntax that needs bash 4:"
grep -n -E "$bash4" $scripts | sed 's/^/ /'
fi

if ((failed)); then
exit 1
fi
echo "The macOS job's scripts ($scripts) pass under bash 3.2."
86 changes: 86 additions & 0 deletions .github/scripts/check-cuda-release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
# Configures the CUDA backend the way users and the release installers get it,
# with ArrayFire's default architecture list and AF_INSTALL_STANDALONE=ON, and
# checks two things that the CI builds' explicit architecture list and plain
# install would hide:
#
# - The default architecture list: prints the flags CMake chose and compiles one
# CUDA object of afcuda with them, which fails when the list holds an
# architecture the toolkit dropped. Without a GPU, the default ("Auto") is the
# common list for the toolkit.
# - The bundled CUDA libraries: installs the cuda_dependencies component into a
# temporary prefix and checks that every library afcu_collect_libs bundles on
# Linux (cuFFT, cuBLAS, cuBLASLt, cuSOLVER, cuSPARSE, nvJitLink, NVRTC and its
# builtins) is there once, under the toolkit's soname, which is the name
# libafcuda loads it by.
#
# Only a configure and one compile: it needs the CUDA toolkit, not a build or a GPU.
#
# Usage: check-cuda-release.sh <source dir> <build dir>
set -euo pipefail

[[ $# -eq 2 ]] || { echo "usage: $0 <source dir> <build dir>" >&2; exit 2; }
source=$1 build=$2
cuda=/usr/local/cuda
failed=

cmake -S "$source" -B "$build" -G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CUDA_COMPILER="$cuda/bin/nvcc" \
-DAF_BUILD_CPU=OFF -DAF_BUILD_CUDA=ON -DAF_BUILD_OPENCL=OFF -DAF_BUILD_ONEAPI=OFF \
-DAF_BUILD_UNIFIED=OFF -DAF_BUILD_FORGE=OFF -DAF_BUILD_EXAMPLES=OFF -DAF_BUILD_DOCS=OFF \
-DBUILD_TESTING=OFF -DAF_WITH_CUDNN=OFF -DAF_INSTALL_STANDALONE=ON > "$build.configure.log" 2>&1 ||
{ tail -n 30 "$build.configure.log"; echo "The configure failed; see $build.configure.log" >&2; exit 1; }
grep -m 1 -E 'GPU detection|common architectures' "$build.configure.log" || true

echo "=== The default architecture list"
object=src/backend/cuda/CMakeFiles/afcuda.dir/cufft.cu.o
flags=$(grep -A 8 "^build $object:" "$build/build.ninja" | grep -m 1 '^ *FLAGS = ' |
grep -o -- '--generate-code=arch=[^ "]*\|-gencode[= ]arch=[^ "]*' | sed 's/^--generate-code=//' || true)
echo "CUDA_architecture_build_targets=$(sed -n 's/^CUDA_architecture_build_targets:[A-Z]*=//p' "$build/CMakeCache.txt")"
echo "Flags for $object:"
while IFS= read -r flag; do echo " $flag"; done <<< "${flags:-(none found)}"
if ninja -C "$build" "$object" > "$build.compile.log" 2>&1; then
echo "Compiling $object with them: passed"
else
grep -E 'nvcc fatal|error' "$build.compile.log" | head -n 5
echo "Compiling $object with them: FAILED" >&2
failed=1
fi

echo "=== The bundled CUDA libraries (AF_INSTALL_STANDALONE=ON, component cuda_dependencies)"
prefix=$(mktemp -d)
if ! cmake --install "$build" --component cuda_dependencies --prefix "$prefix" > "$build.install.log" 2>&1; then
grep -E 'CMake Error|cannot find|not found' "$build.install.log" | head -n 5
echo "Installing cuda_dependencies: FAILED" >&2
failed=1
fi
soname() { { readelf -d "$1" 2> /dev/null || true; } | sed -n 's/.*(SONAME).*\[\(.*\)\]/\1/p'; }
printf '%-15s %-28s %-28s %s\n' library "toolkit soname" bundled "bundled file's soname"
for lib in cufft cublas cublasLt cusolver cusparse nvJitLink nvrtc nvrtc-builtins; do
expected=$(soname "$(readlink -f "$cuda/lib64/lib$lib.so")")
expected=${expected:-"(no lib$lib.so)"}
bundled=$(cd "$prefix" && find . -name "lib$lib.so*" | sed 's|^\./||' | sort | tr '\n' ' ')
bundled=${bundled% }
if [[ -z $bundled ]]; then
printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "MISSING" "-"
failed=1
elif [[ $bundled == *' '* ]]; then
printf '%-15s %-28s %-28s %s\n' "$lib" "$expected" "$bundled" "(more than one)"
failed=1
else
actual=$(soname "$prefix/$bundled")
status=
[[ ${bundled##*/} == "$expected" && $actual == "$expected" ]] || { status=" <- WRONG NAME"; failed=1; }
printf '%-15s %-28s %-28s %s%s\n' "$lib" "$expected" "$bundled" "$actual" "$status"
fi
done
others=$(cd "$prefix" && find . -type f | sed 's|^\./||' |
grep -v -E '/lib(cufft|cublas|cublasLt|cusolver|cusparse|nvJitLink|nvrtc|nvrtc-builtins)\.so' || true)
[[ -z $others ]] || echo "Also bundled: $others"
rm -rf "$prefix"

if [[ -n $failed ]]; then
echo "The release configuration check failed." >&2
exit 1
fi
echo "The release configuration check passed."
Loading