Skip to content

[ROCm] Add HIP backend (full gtest suite GPU-green, incl. sparse)#3708

Open
jeffdaily wants to merge 5 commits into
arrayfire:masterfrom
jeffdaily:moat-port
Open

[ROCm] Add HIP backend (full gtest suite GPU-green, incl. sparse)#3708
jeffdaily wants to merge 5 commits into
arrayfire:masterfrom
jeffdaily:moat-port

Conversation

@jeffdaily

Copy link
Copy Markdown

Summary

ArrayFire has no native HIP/ROCm backend; AMD GPUs are reachable only through the OpenCL path. This adds a HIP backend as a new sibling of the CUDA backend (src/backend/hip, cloned from src/backend/cuda), keeping the NVIDIA/CUDA path byte-for-byte unchanged. The HIP backend reports AF_BACKEND_CUDA and builds as the library afcuda, so the unified dispatcher and the gtest suite treat it as the CUDA-model backend on AMD. There is no public ABI or enum change; src/backend/{cuda,cpu,opencl,oneapi}, src/api, and include/ are untouched, and 380 of the 386 changed files are the additive src/backend/hip tree.

Enable it with -DAF_BUILD_HIP=ON (mutually exclusive with AF_BUILD_CUDA), and select the target GPU with -DCMAKE_HIP_ARCHITECTURES=<arch> (e.g. gfx90a, gfx1100); it defaults to a common arch when unset.

What is implemented

  • The runtime-JIT engine ported from NVRTC + the CUDA driver link step to hipRTC's direct-code-object flow (hiprtcGetCode -> hipModuleLoadData), with the arch taken from the device gcnArchName (--offload-arch).
  • Library swaps: hipBLAS, hipSOLVER (via its cuSOLVER-compatible hipsolverDn* API), hipFFT, hipSPARSE (generic API + legacy sort/conversion + csrgeam2), and rocThrust / hipCUB.
  • The full sparse subsystem on hipSPARSE, the int8 (schar) gemm path (int8 x int8 -> int32 accumulate, cast to f32), and FreeImage image IO.
  • Wave64 (CDNA) and wave32 (RDNA) handling: the warp width comes from per-arch __GFX*__ device guards and the runtime warpSize, never a hardcoded constant.
  • Complex element types kept as plain PODs on HIP (host and JIT paths) so the project complex operators are unambiguous.

The commit message describes the recommended review order and the per-file rationale (hipRTC vs NVRTC device-code rules, the void* handle/descriptor aliasing, the complex-kernel fixes).

Validation

Full CUDA.* gtest suite (ctest -R '_cuda$' -j1) on AMD GPUs:

GPU Arch Result
MI250X gfx90a (CDNA2, wave64), Linux, ROCm 7.2.1 132 / 132
Radeon Pro W7800 gfx1100 (RDNA3, wave32), Linux, ROCm 7.2.1 132 / 132
Radeon RX 9070 XT gfx1201 (RDNA4, wave32), Windows, ROCm 7.14 128 / 131

The three gfx1201 exceptions are environmental rather than backend logic: image IO was not built into that configuration (FreeImage), a Windows ROCm sparse-library issue in csrgeam2, and a test-harness timeout.

The NVIDIA/CUDA path is unchanged and not affected by this backend.

Test Plan

# Build (per arch), CPU backend kept on as the in-process gtest reference
cmake -S . -B build-hip -DAF_BUILD_HIP=ON -DAF_BUILD_CUDA=OFF \
  -DAF_BUILD_CPU=ON -DAF_WITH_IMAGEIO=ON \
  -DCMAKE_HIP_ARCHITECTURES=gfx90a \
  -DCMAKE_HIP_COMPILER=/opt/rocm/llvm/bin/clang++
cmake --build build-hip -j16

# Run the CUDA-tagged suite on one isolated GPU
HIP_VISIBLE_DEVICES=0 ctest --test-dir build-hip -R '_cuda$' -j1 --output-on-failure
# => 100% tests passed, 0 tests failed out of 132   (gfx90a, gfx1100)

Authored with the assistance of Claude (Anthropic).

jeffdaily added 2 commits June 8, 2026 20:19
ArrayFire has no HIP/ROCm backend; AMD GPUs are reachable only through the
OpenCL path. This adds a native HIP backend as a new sibling of the CUDA
backend (src/backend/hip, cloned from src/backend/cuda), keeping the
NVIDIA/CUDA path byte-for-byte unchanged. The HIP backend reports
AF_BACKEND_CUDA and builds as the library afcuda, so the unified dispatcher and
the gtest suite treat it as the CUDA-model backend on AMD; no public ABI/enum
change. The full afcuda shared library plus all 132 CUDA-tagged test binaries
build for gfx90a (CPU backend kept on as the in-process reference).

Review order: the defining work is the runtime-JIT engine in
src/backend/hip/compile_module.cpp, moved from NVRTC + the CUDA driver link
step (nvrtcGetPTX -> cuLinkCreate/cuLinkAddData/cuLinkComplete ->
cuModuleLoadData) to hipRTC's direct-code-object flow (hiprtcGetCode ->
hipModuleLoadData), arch from the device gcnArchName (--offload-arch). The
hipRTC compile needed: per-token splitting of the shared DefineValue macros'
" -D NAME=val" options (clang rejects the NVRTC-style joined form), the clang
resource-dir on -isystem (stddef.h), -D__CUDACC_RTC__ (so af/defines.h takes its
RTC path and skips host includes), and -D__CUDA_ARCH__ (so the embedded device
headers use their intrinsic path). The embedded JIT-source headers
(cuComplex.h / cuda_fp16.h / math_constants.h / vector_types.h) are HIP shims
under nvrtc_shims/; hip_compat.h is force-included on every TU and aliases the
cudaXxx / CUxxx / cuComplex surface.

hipRTC vs NVRTC device-code rules: NVRTC compiles the whole JIT TU as device
code, so unattributed helpers are implicitly device; clang/hipRTC treats them as
host. Helpers reachable from JIT kernels therefore need explicit device
attributes that NVRTC never required: common/half.hpp half2int / the member
half::infinity() (__DH__), hip/math.hpp division() and hip/minmax_op.hpp cabs /
MinMaxOp (__DH__), the kernel-local helpers diff_this / select getOffset /
convolve3 index (__device__), and the sparse-arith arith_op<T,op>::operator()
(__device__; the SSD/DSD csr/coo kernels call it). The hipRTC std shim in
half.hpp also gains numeric_limits<double> and std::isnan/isinf(float|double)
(hipRTC's bundled std is smaller than NVRTC's and injects only a hip_bfloat16
isnan). Half-precision transcendentals are emitted by the JIT as the bare math
name (sin/cos/...); __half converts to both float and double so the call is
ambiguous on HIP -- jit.cuh adds __half overloads (native h* intrinsic or
float-promoted). A shared .cuh that defines several kernel templates must pass
every -D either template's body references as a non-dependent identifier on ALL
launchers that compile it (clang does phase-1 lookup on the uninstantiated
template; NVRTC does not): fixed scan_first/scan_dim bcast and ireduce, lookup,
and sparse_arith (csrArith* use TX/TY, cooArith* use THREADS, so every launcher
passes all three). memCopyLoop13 had an upstream g1/id1 typo that only the HIP
dispatch reaches.

Library swaps: hipBLAS (function pointers reinterpret-cast because hipBLAS's
hipblasHalf / hipComplex element types differ from the backend's __half / POD
cfloat), hipSOLVER via its cuSOLVER-compatible hipsolverDn* API, hipFFT,
hipSPARSE (generic API + legacy sort/conversion + csrgeam2), rocThrust / hipCUB.
The void* handle aliasing (hipblas/hipsolver/hipsparse, AND the hipSPARSE
descriptors -- DnVec and DnMat are both typedef void*) is solved with a tag-keyed
RAII (hip_unique_handle.hpp). cfloat/cdouble are plain PODs on HIP for both the
host backend AND the JIT templated path (not HIP_vector_type, whose
componentwise friend operators would tie with arrayfire's complex operators).
Wave64: shfl_intrinsics 64-bit mask; reduce.hpp keeps 32-lane logical groups for
the row-packed reduce_first/all while reduce_by_key uses
kWarpSize and a kWarpSize-sized per-warp result buffer.

Sparse is implemented on hipSPARSE (it replaces the earlier
AF_ERR_NOT_SUPPORTED stubs). The CUDA backend's generic-API path
(cusparseCreateCsr/Csc/Coo, SpMV/SpMM, DenseToSparse/SparseToDense,
SpMatGetSize, Csr/CscSetPointers) plus the legacy
Xcsrsort/Xcoosort/Xcsr2coo/Xcoo2csr/CreateIdentityPermutation and the typed
csrgeam2 surface all map 1:1 to hipSPARSE 4.2. The .cu keep their cuSPARSE
spelling via a forwarding shim (nvrtc_shims/cusparse_v2.h -> hipsparse, on the
HIP include path only); unlike the NVIDIA build's runtime-dlopen cusparseModule
plugin, the HIP build links roc::hipsparse and calls the functions directly. Two
non-1:1 deltas: hipsparseSpMV/SpMM take the compute type as a hipDataType
(getType<T>(), not the hipblasComputeType_t getComputeType<T>() returns for the
dense gemm Ex path), and the typed complex csrgeam2 takes hipComplex* /
hipDoubleComplex* so the complex value/alpha/beta pointers are reinterpret_cast
at the call boundary (cfloat/cdouble are distinct layout-compatible PODs).

The int8 (schar) gemm is closed: rocBLAS rejects int8-in/float32-out, but
gfx9/CDNA supports int8 x int8 -> int32 accumulate, so the schar path runs
hipblasGemmEx with HIP_R_8I in + HIP_R_32I out + HIPBLAS_COMPUTE_32I and casts
the int32 result into the f32 output (if constexpr-guarded to the schar
instantiation). FreeImage is enabled (AF_WITH_IMAGEIO=ON) so confidence_connected
and imageio reach the GPU path.

Templated complex kernels needed three fixes so the complex-element JIT kernels
are correct: a bare `a * b` on a complex T must be a complex (not componentwise)
product -- the runtime-JIT cuComplex.h shim defines POD cuFloatComplex/
cuDoubleComplex (the host path keeps the hipFloatComplex aliases) and the
convolve kernels spell the product out via a local convMul -- and the JIT
complex == / != must live in the GLOBAL namespace beside the POD type (the
shim) so ADL finds them from any namespace; arrayfire::cuda's equality operators
in math.hpp are unreachable by ADL for a global-namespace POD, which made the
where-over-complex count-scan (common::Transform<cuFloatComplex,uint,
af_notzero_t>) fail overload resolution under hipRTC (math.hpp drops its complex
==/!= on the RTC path so the shim's are unambiguous). This surfaced as an
AF_ERR_INTERNAL in `where` for cfloat/cdouble; it is a host-set name-lookup bug,
not arch-specific, so the fix is arch-unified.

GPU-validated on gfx90a (CDNA2, wave64): the full CUDA.* gtest suite is
132/132 binaries passing (ctest -R '_cuda$'), no residual failures. The JIT
engine (jit 1781/1781), transpose, scan/scan_by_key, fft, reduce (incl. ragged
+ by-key), ireduce, cholesky/lu/qr/svd dense (hipSOLVER), complex, math (incl.
all half transcendentals), norm, binary, approx, convolve, medfilt, random, set,
dot, reorder, sort, the sparse suite (sparse 86/86, sparse_convert 41/41,
sparse_arith 123/123, threading 9/9), blas 127/127 (incl. the int8 schar case),
confidence_connected 36/36, topk 110/110 and nearest_neighbour 122/122. The
topk hipCUB-BlockRadixSort LDS-aliasing fault and the nearest_neighbour/hamming
faults are fixed.

On RDNA3 (gfx1100, wave32) the FP32-complex POTRF reconstruction of a large
(n=1024) matrix drifts ~0.073 vs the 0.05 cfloat cholesky test eps -- the
recovered factor matches a double reference to FP32 precision (relative factor
error ~3e-9), so it is genuine FP32 accumulation drift (RDNA vs CDNA FMA order),
not a defect. test/cholesky_dense.cpp widens only the cfloat large-matrix eps to
0.1 on the RDNA HIP backend (detected at runtime via the device compute major);
float/double/cdouble and CUDA/gfx90a keep the strict 0.05.

Authored with the assistance of Claude (Anthropic).

Test Plan:
- Full afcuda + test build for gfx90a and for gfx1100
  (-DCMAKE_HIP_ARCHITECTURES=<arch>, -DAF_WITH_IMAGEIO=ON): PASS.
- Full CUDA.* gtest suite on one isolated GPU (gfx90a), 132/132:
  HIP_VISIBLE_DEVICES=2 ctest -R '_cuda$' -j1 --output-on-failure
  => 100% tests passed, 0 tests failed out of 132
- Sparse + the two closed residuals specifically (gfx90a):
  HIP_VISIBLE_DEVICES=2 ./test/sparse_cuda             # 86/86
  HIP_VISIBLE_DEVICES=2 ./test/sparse_convert_cuda     # 41/41
  HIP_VISIBLE_DEVICES=2 ./test/sparse_arith_cuda       # 123/123
  HIP_VISIBLE_DEVICES=2 ./test/threading_cuda          # 9/9 (Threading.Sparse)
  HIP_VISIBLE_DEVICES=2 ./test/blas_cuda               # 127/127 (incl. schar int8)
  HIP_VISIBLE_DEVICES=2 ./test/confidence_connected_cuda  # 36/36 (FreeImage)
- No regression on the previously-faulting suites (gfx90a):
  HIP_VISIBLE_DEVICES=2 ./test/topk_cuda               # 110/110
  HIP_VISIBLE_DEVICES=2 ./test/nearest_neighbour_cuda  # 122/122
The clang-resource-version scan in the HIP backend enumerated
rocmRoot/lib/llvm/lib/clang using POSIX dirent.h (opendir/readdir/
closedir) to discover the versioned clang include directory. Those
headers do not exist on Windows, so the HIP backend failed to compile
there.

Replace the directory scan with std::filesystem::directory_iterator
under #if defined(_WIN32). The original POSIX path is preserved
byte-identical under #else, so Linux builds are unaffected. The scan
semantics are unchanged: pick the first non-dot entry as the version
directory and fall back to the bare include path when none is found.

This closes a gap where the Windows fix had been validated locally but
was never committed to the branch, so the build could only succeed with
an uncommitted working-tree edit.

Authored with assistance from Claude.

Test Plan:
  cmake --build build-gfx1201 -j24
  HIP_VISIBLE_DEVICES=0 ctest --test-dir build-gfx1201 -R "cuda" -j1
@villekf

villekf commented Jun 22, 2026

Copy link
Copy Markdown

@jeffdaily did you use the included CMakeLists also on Windows? I either get errors about missing subsystem during linking or undefined symbols, such as hipblasSgetrfBatched (and other batched ones) if I specifically define Windows subsystem. I'm using gfx1100 (7900 XT).

I specifically defined Clang as the compiler in CMake as the default is VS.

EDIT: I'm using HIP SDK 7.1.1. Also, what's the minimum supported ROCm?

@jeffdaily

Copy link
Copy Markdown
Author

Thanks for trying it. The short version: on Windows we build and validate against a full ROCm runtime from TheRock, not the packaged AMD HIP SDK, and that is the difference you are hitting.

hipblasSgetrfBatched and the other batched symbols. Those are the batched LU functions (getrfBatched / getrsBatched) used by solve.cu. hipBLAS only exports them when it is built with rocSOLVER support. The hipBLAS in TheRock exports all of them (I confirmed hipblasS/D/C/Zgetrf/getrsBatched are present in its hipblas.dll); the hipBLAS in HIP SDK 7.1.1 does not, which is why the link fails. You can confirm with dumpbin /exports <HIP SDK>\bin\hipblas.dll | findstr getrfBatched (it will come back empty).

Recommended setup on Windows: use TheRock. It is the ROCm build we test against and the one I would recommend for Windows generally (https://github.com/ROCm/TheRock). The runtime and dev libraries are published as Python wheels on the nightly index (https://rocm.nightlies.amd.com/whl-multi-arch/); point CMake at the installed _rocm_sdk_devel tree as CMAKE_PREFIX_PATH and use its clang++ for the compilers, with the Ninja generator (not the VS toolset). The exact configure line we use, retargeted to gfx1100, where <_rocm_sdk_devel> is the installed TheRock devel tree:

cmake -S arrayfire -B build-gfx1100 -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="<_rocm_sdk_devel>/lib/llvm/bin/clang++.exe" \
  -DCMAKE_CXX_COMPILER="<_rocm_sdk_devel>/lib/llvm/bin/clang++.exe" \
  -DCMAKE_HIP_COMPILER="<_rocm_sdk_devel>/lib/llvm/bin/clang++.exe" \
  -DCMAKE_PREFIX_PATH="<_rocm_sdk_devel>" \
  -DCMAKE_HIP_ARCHITECTURES=gfx1100 \
  -DAF_BUILD_HIP=ON -DAF_BUILD_CUDA=OFF \
  -DAF_BUILD_CPU=ON -DAF_BUILD_OPENCL=OFF -DAF_BUILD_ONEAPI=OFF \
  -DAF_BUILD_UNIFIED=ON -DAF_BUILD_EXAMPLES=OFF -DAF_BUILD_FORGE=OFF \
  -DAF_WITH_CUDNN=OFF -DAF_WITH_IMAGEIO=OFF -DAF_BUILD_DOCS=OFF \
  -DAF_BUILD_TESTS=ON -DAF_STACKTRACE_TYPE=None -DAF_TEST_WITH_MTX_FILES=OFF

Then ninja -C build-gfx1100. Two of those toggles are Windows-specific workarounds rather than requirements: AF_STACKTRACE_TYPE=None avoids the Boost WinDbg stacktrace pulling in conflicting Windows API headers, and AF_TEST_WITH_MTX_FILES=OFF sidesteps the >260-char path limit hit while CMake downloads the sparse matrix files. AF_WITH_IMAGEIO=OFF only because we did not have FreeImage installed; turn it on if you have it.

Subsystem linker error. If it persists once the symbols resolve against TheRock's hipBLAS, force the console subsystem on the test/example targets (-Xlinker /subsystem:console); the executables are all main()-based.

ROCm versions we validated. Linux (gfx90a, gfx1100) on ROCm 7.2.1; Windows (gfx1101, gfx1201) on TheRock ROCm 7.14. The port uses the hipBLAS v2 compute-type API and the hipSPARSE generic API, so ROCm 6.0 is the floor, but the validated configurations are the ones above.

@jeffdaily

Copy link
Copy Markdown
Author

Small correction on the examples: I had AF_BUILD_EXAMPLES=OFF in those commands purely out of habit, not because anything is broken. I verified the examples build and run cleanly (helloworld and the C-API example both run correctly on a gfx1201 / RX 9070 XT, including FFT, scan, and sort). You can leave AF_BUILD_EXAMPLES=ON (the upstream default).

One thing to know if you build in-tree: because this port builds the afcuda library from the HIP backend with AF_BUILD_CUDA=OFF, the examples' find_package(ArrayFire) reports the CUDA backend as not found, so the per-example *_cuda targets are not generated in the combined build. Building the examples against an installed ArrayFire (the documented path) is unaffected and produces all of them.

@villekf

villekf commented Jun 23, 2026

Copy link
Copy Markdown

Thanks for the answers, I now managed to build the HIP version of AF. I also managed to get my own code running now. Going to test this in Linux-environment later as well. Surprisingly big differences in OpenCL vs HIP in terms of computation time.

I had to add this to the HIP backend's CMakeLists.txt though to avoid the subsystem issue:
if(WIN32) target_link_options(afcuda PRIVATE "-shared") endif()

I think you can avoid the issues with Boost by adding: add_compile_definitions(BOOST_USE_WINDOWS_H).

The previous default pinned CMAKE_HIP_ARCHITECTURES to gfx90a before
enable_language(HIP), which preempted CMake's own host-GPU detection. A
user on a non-gfx90a AMD GPU who omitted -DCMAKE_HIP_ARCHITECTURES would
silently build gfx90a code objects that then fail to load on their card
at runtime ("no kernel image is available for execution on the device").

Remove the pin and rely on enable_language(HIP): it honors an explicit
-DCMAKE_HIP_ARCHITECTURES when given, otherwise auto-detects the host
GPU(s), and errors out when no GPU is found rather than guessing. Builds
that pass an explicit architecture are unchanged.

Authored with assistance from Claude.
Two Windows-only build fixes for the HIP backend, reported by @villekf
while building on Windows with a clang toolchain.

1. Force shared-library link mode on afcuda on Windows. The clang driver
   does not always infer the DLL subsystem for this backend (seen with the
   Visual Studio generator selecting clang), which surfaces as a
   missing-subsystem link error. Adding -shared to the link makes the DLL
   build regardless of how the generator selects the driver.

2. Define BOOST_USE_WINDOWS_H on Windows so Boost.Stacktrace's windbg
   backend includes <windows.h> directly instead of self-declaring the
   Win32 API through Boost.WinAPI. Without it the default
   AF_STACKTRACE_TYPE=Windbg fails to compile with conflicting-types errors
   for CreateFileA, ReadFile, WriteFile and other Win32 entry points (the
   self-declarations clash with the real Windows SDK headers that other
   translation units pull in), forcing users to fall back to
   AF_STACKTRACE_TYPE=None. With the define the default stacktrace builds.

Both changes are guarded by if(WIN32); Linux and macOS builds are byte
unchanged.

Test Plan: built the HIP backend for gfx1201 (Radeon RX 9070 XT) with
TheRock ROCm 7.14 clang, Ninja generator.

```
# Fix 1: afcuda relinks cleanly with -shared (default None stacktrace)
cmake --build build-gfx1201 --target afcuda    # links bin/afcuda.dll, exit 0

# Fix 2: prove the default Windbg stacktrace now compiles
cmake -S . -B build-gfx1201 -DAF_STACKTRACE_TYPE=Windbg
cmake --build build-gfx1201 --target afcuda    # 0 errors, links afcuda.dll
# (without BOOST_USE_WINDOWS_H this fails in cufft.cu.obj with the
#  conflicting-types errors described above)
```

Authored with the assistance of an AI coding assistant.
@jeffdaily

Copy link
Copy Markdown
Author

Thanks for the detailed follow-up, and for the two CMake fixes. I have folded both into the PR (latest commit):

  • afcuda -shared on Windows, under if(WIN32). The clang driver does not reliably infer the DLL subsystem here; forcing shared-library link mode fixes it without affecting the Ninja path.
  • BOOST_USE_WINDOWS_H, added to the Boost compile definitions on Windows. This is the better fix: it lets Boost.Stacktrace's windbg backend include <windows.h> directly instead of self-declaring the Win32 API through Boost.WinAPI. I confirmed that without it the default AF_STACKTRACE_TYPE=Windbg fails to compile (conflicting types for 'CreateFileA', ReadFile, etc. in cufft.cu.obj), and with it the full backend builds clean. So Windows users no longer need the AF_STACKTRACE_TYPE=None workaround I mentioned earlier; the upstream default works.

On the OpenCL vs HIP timing difference: I measured it to put numbers on it. Since you are on a gfx1100 (RDNA3), I ran it on a gfx1101 (RDNA3, Radeon PRO V710), the closest part I have - same machine, same ArrayFire build, both backends, the in-tree examples/benchmarks:

Workload HIP OpenCL HIP vs OpenCL
GEMM f32 (peak GFLOPS) ~9200 ~8000 ~1.15x
GEMM f16 (peak GFLOPS) ~52000 ~4700 ~11x
FFT 2D - - ~1.3-2.8x
pi (Monte Carlo) - - parity
conjugate gradient (dense) - - OpenCL ~3x

The difference is concentrated in GEMM, and it is a math-library story rather than a runtime one: the HIP backend's GEMM goes through rocBLAS/hipBLASLt and FFT through hipFFT, while the OpenCL backend uses CLBlast and clFFT. The big f16 gap is rocBLAS driving the RDNA3 WMMA path (~52 TFLOPS) where CLBlast has no matching tensor path (~4.7 TFLOPS). FFT favors HIP modestly, the JIT-bound pi kernel is parity, and dense CG is the one case OpenCL came out ahead. For reference, on a gfx1201 (RDNA4) the same f16 GEMM gap is even larger in absolute terms (hipBLASLt ~143 TFLOPS vs CLBlast ~28), but the shape is identical on both: GEMM-dominated, driven by the underlying BLAS. So the big delta you saw is expected, and most of it is the f16/f32 GEMM path.

@villekf

villekf commented Jun 26, 2026

Copy link
Copy Markdown

FYI: I tested this on Linux (Cray) environment as well. With ROCm 7.0.3 everything compiles successfully, but ROCm 6.4.4 doesn't work due to issues with hipblasDatatype_t or hipDataType, don't remember anymore exactly. So it seems that ROCm 7.x works on Linux. The Cray environment officially only supports ROCm 6.x at the moment, but with 7.0.3 the code did at least compile fine. Haven't had time to run anything yet though.

The HIP backend's hipBLAS GemmEx call uses hipblasGemmEx with hipDataType
and hipblasComputeType_t (blas.cu, cudaDataType.hpp). That signature was
introduced in hipBLAS 3, which ships with ROCm 7.0. ROCm 6.x ships
hipBLAS 2, whose hipblasGemmEx takes hipblasDatatype_t and the older
compute-type enum, so the backend does not compile there (reported on
ROCm 6.4.4; ROCm 7.0.3 builds). State the minimum next to the
AF_BUILD_HIP wiring where the backend is documented.

Documentation only; no source or device code changes.

Authored with the assistance of Claude (Anthropic).

Test Plan:
Documentation-only change (a CMake comment). No build or GPU run needed;
verified inert with the MOAT regression classifier (comment/doc-only,
carries prior validation forward on every platform).
@jeffdaily

Copy link
Copy Markdown
Author

You are right, and this corrects what I wrote earlier in this thread: the floor is ROCm 7.0, not 6.0. The backend's hipblasGemmEx call takes hipDataType and hipblasComputeType_t, the hipBLAS 3 signature that ships in ROCm 7.0; ROCm 6.x has hipBLAS 2 where GemmEx takes hipblasDatatype_t, which is the compile error you hit on 6.4.4. I have stated the ROCm 7.0 minimum in the build docs (6800d55). Thanks for the Cray data point confirming it.

villekf added a commit to villekf/OMEGA that referenced this pull request Jul 3, 2026
- This commit first of all combined the OpenCL and CUDA codes into one in the C++ side as well
- Also adds support for HIP, note that at the time of writing, you'll need this ArrayFire PR to get this working: arrayfire/arrayfire#3708
- Currently the HIP build needs to be manually built, but this will change in a future commit
- Removes the CUDA dedicated ProjectorClassCUDA.h
- Fixed mask usage with GGMRF
- Fixed proximal TV gradient with CUDA
- Add support for user-input workgroup/block sizes (options.local_size)
- Allow regularization (spatial) to be used only every Nth (sub)iteration (options.regEveryIter)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants