# Math support for GPU kernels

**URL:** <https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381>\
**Category:** GPU\
**Tags:** amdgpu\
**Created:** [April 1, 2024, 5:18pm UTC](https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381 "2024-04-01T17:18:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fwinter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fwinter/32/208193_2.png) [@fwinter](https://discourse.julialang.org/u/fwinter)\
**Post date:** [April 1, 2024, 5:18pm UTC](https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381/1 "2024-04-01T17:18:01Z")

</div>

I was hoping you could provide some insight into the process of incorporating support for mathematical functions (such as sin, cos, sqrt, etc.) in GPU kernels.

Within the ROCm installation, there exists a file located at ‘/opt/rocm-6.0.0/llvm/lib/libomptarget-old-amdgpu-gfx90a.bc’, which contains mathematical functions. For AMDGPU support in Julia, is it customary to link this file to the IR module?

My curiosity stems from the presence of the OpenLibm project ([openlibm.org](http://openlibm.org)). However, I presume that this project is not utilized for mathematical support within GPU kernels, correct?

---

<div class="post-metadata">

**Author:** ![pxl-th](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pxl-th/32/31939_2.png) [@pxl-th](https://discourse.julialang.org/u/pxl-th)\
**Post date:** [April 1, 2024, 5:38pm UTC](https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381/2 "2024-04-01T17:38:03Z")

</div>

Yes, we currently always link device libraries when compiling GPU kernels.  
You can also see where we define support for math functions [here](https://github.com/JuliaGPU/AMDGPU.jl/blob/master/src/device/gcn/math.jl).

Actually I haven’t seen openlibm, but we are using device libraries for other things so it makes sense to use them.

---

<div class="post-metadata">

**Author:** ![fwinter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fwinter/32/208193_2.png) [@fwinter](https://discourse.julialang.org/u/fwinter)\
**Post date:** [April 1, 2024, 6:44pm UTC](https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381/3 "2024-04-01T18:44:11Z")

</div>

Thank you for providing clarification. Could you please direct me to the specific code where the reading of the device library occurs? I attempted to search within AMDGPU.jl but was unable to locate where a file such as ‘libomptarget’ would be accessed. Alternatively, do you package an intermediate representation (IR) version of the libdevice and forego the need to locate the device library at runtime via ROCM\_PATH?

A bit of background: Within my library [qdp-jit](https://github.com/JeffersonLab/qdp-jit), I dynamically generate GPU kernels using LLVM. Kernels incorporating mathematical functions are linked to the libdevice bundled with the SDK (CUDA/ROCm). While this approach functions smoothly for CUDA, an issue has arisen with ROCm since version 5.5.1, where the bundled device library ceases to function properly. I am currently exploring how colleagues address this challenge.

---

<div class="post-metadata">

**Author:** ![pxl-th](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pxl-th/32/31939_2.png) [@pxl-th](https://discourse.julialang.org/u/pxl-th)\
**Post date:** [April 1, 2024, 8:15pm UTC](https://discourse.julialang.org/t/math-support-for-gpu-kernels/112381/4 "2024-04-01T20:15:52Z")

</div>

Reading of device libraries occurs [here](https://github.com/JuliaGPU/AMDGPU.jl/blob/26877501ccf2b970fe3e4a44bd730fb2aef99145/src/compiler/device_libs.jl#L27), which is called during one of the [compilation stages](https://github.com/JuliaGPU/AMDGPU.jl/blob/26877501ccf2b970fe3e4a44bd730fb2aef99145/src/compiler/codegen.jl#L50).

We do [locate ROCm libraries](https://github.com/JuliaGPU/AMDGPU.jl/blob/26877501ccf2b970fe3e4a44bd730fb2aef99145/src/discovery/discovery.jl#L97) during runtime, however, there are some exceptions.  
At some point we were able to ship almost all of the ROCm stack as artifacts via [Binary Builder](https://binarybuilder.org/).  
Meaning the users wouldn’t need to install ROCm manually and Julia’s package manager would handle the installation.  
However, it is too bothersome to maintain build recipes so for now we again require users to manually install ROCm.

I don’t recall having issues with 5.5 device libraries, but we had an issue where newer ROCm versions started using LLVM 16+, but Julia was still at LLVM 15 and we couldn’t read newer bitcode with older LLVM.  
To fix that we’ve [downgraded bitcode files](https://github.com/JuliaPackaging/Yggdrasil/tree/master/R/ROCmDeviceLibs/ROCmDeviceLibs%405.6.1/bundled/patches) to an older version of LLVM and are shipping them via BinaryBuilder as artifacts.