Our app logs GPU temperature (AMDSMI_TEMPERATURE_TYPE_EDGE, AMDSMI_TEMP_CURRENT), fan speed and VRAM usage using AMD SMI API, none were over the limits when the fault occured (following are the last measurements before the segfault):

GPU 0/Numa node 0: 79°C, fan 29%, VRAM 16%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 11%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 13%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 10%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 11%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 13%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 10%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 9%. 
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 15%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 14%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 12%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 10%.
GPU 0/Numa node 0: 79°C, fan 29%, VRAM 14%.

stderr logged:
Memory access fault by GPU node-2 (Agent handle: 0x10306d0) on address (nil). Reason: Page not present or supervisor privilege.

By the nature of the program there is very high traffic over the PCIEx bus, most of the time is spent on this, because new data must be constantly uploaded, processed into new VRAM buffers and then downloaded back to the host, at which point all these buffers are freed. Two kernels process each chunk of data and they run 10% of the time needed to move data up and down the GPU.
Judging by the VRAM usage only, there seems to be no memory leak? If there was one, it would manifest much earlier given the high quantity of data that is moved between the host and the GPU.
We use callbacks to notify when a chunk of data is finished processing and downloaded to the host. I noticed that every time the segfault occured, the host code was waiting for the callback to be called.

Managed to export for our app the env var HSAKMT_DEBUG_LEVEL to 7. This is what stderr logged when the fault occured (unrelated lines were removed, most were alloc/map/free/unmap):

op get range attrs failed Bad address
Address does not belong to a known buffer
Memory access fault by GPU node-2 (Agent handle: 0x1068a50) on address 0x34000. Reason: Page not present or supervisor privilege.

The address looks like a local address, all the other the addresses when alloc/map/free/unmap operations are logged look like 0x7fcx xxxx xxxx
I'm going to try a debug build hoping to squeeze a little more info.

I got no additional info by reproducing with a debug build of ours.
The HSA stack doesn't seem to indicate the source of the problem.
Is there a way to tell whether some OpenCL kernel is the culprit or if there even was a kernel running at the time the fault occured?
The stack trace only shows that the host code was waiting for the callback set to the kernels' event completions to be called.

Hi @claudiubalogh, maybe try using rocprofv3 https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/amd-mainline/how-to/using-rocprofv3.html for --hsa-trace --kernel-trace, and --memory-allocation-trace? Hopeful these will reveal more information.

Unfortunately rocprofv3 / rocm 6.3.3 does not recognize --memory-allocation-trace:

rocprofv3: error: unrecognized arguments: --memory-allocation-trace

Upgrading to a newer rocm is not possible yet, I must use OpenSuse 15.5 with linux kernel:

5.14.21-150500.55.65-default #1 SMP PREEMPT_DYNAMIC Thu May 23 04:57:11 UTC 2024 (a46829d) x86_64 x86_64 x86_64 GNU/Linux
I'll run with the other traces.