# ROCm-Device-Libs BinaryBuilder
**URL:** https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962
**Category:** Tooling
**Tags:** binarybuilder, amdgpu, rocm
**Created:** [July 8, 2022, 5:41pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962 "2022-07-08T17:41:22Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@pxl-th](https://discourse.julialang.org/u/pxl-th)
#### Post date: [July 8, 2022, 5:41pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/1 "2022-07-08T17:41:22Z")
I’m having trouble building ROCm-Device-Libraries using BinaryBuilder with ROCmLLVM.
I’m trying to follow build recipe from the [rocm-arch](https://github.com/rocm-arch/rocm-arch/blob/98c92d1bb9fec141bc442f39d8a6fc40439a92cb/rocm-device-libs/PKGBUILD) repository.
There they use ROCmLLVM to build it. But when specifying it in the BinaryBuilder I get errors I’m not sure show to fix.
The gist of the error:
```bash
The C compiler
"/workspace/destdir/tools/clang"
is not able to compile a simple test program.
It fails with the following output:
...
/workspace/destdir/tools/clang CMakeFiles/cmTC_7b11b.dir/testCCompiler.c.o -o cmTC_7b11b
/opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crt1.o: No such file or directory
/opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crti.o: No such file or directory
/opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crtbegin.o: No such file or directory
/opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find -lgcc
/opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find -lgcc_s
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
```
I’ve tried specifying toolchain file `${CMAKE_TARGET_TOOLCHAIN}` as well as clang specific `${CMAKE_TARGET_TOOLCHAIN%.*}_clang.cmake`.
But they overwrite the compiler with their own `/opt/bin/x86_64-linux-gnu-libgfortran5-cxx11/x86_64-linux-gnu-clang`
Am I missing somethings that needs to be configured when using different compiler?
I want to compile all the necessary packages to be able to build rocBLAS while reducing number of patches that disable features and potentially avoid [mixing musl and glibc](https://github.com/JuliaPackaging/Yggdrasil/pull/4255#issuecomment-1172900653) in the process.
Here’s the recipe:
```julia
using Pkg
using BinaryBuilder
name = "ROCmDeviceLibs"
version = v"4.2.0"
sources = [
ArchiveSource(
"https://github.com/RadeonOpenCompute/ROCm-Device-Libs/archive/rocm-$(version).tar.gz",
"34a2ac39b9bb7cfa8175cbab05d30e7f3c06aaffce99eed5f79c616d0f910f5f"),
]
script = raw"""
cd ${WORKSPACE}/srcdir/ROCm-Device-Libs*/
mkdir build && cd build
ln -s ${prefix}/bin/clang ${prefix}/tools/clang
ln -s ${prefix}/bin/lld ${prefix}/tools/lld
export PATH="${prefix}/bin:${prefix}/tools:${PATH}"
CC=${prefix}/tools/clang \
CXX=${prefix}/tools/clang++ \
cmake \
-DCMAKE_PREFIX_PATH=${prefix} \
-DCMAKE_INSTALL_PREFIX=${prefix} \
-DLLVM_DIR="${prefix}/lib/cmake/llvm" \
-DClang_DIR="${prefix}/lib/cmake/clang" \
..
make -j${nproc}
make install
"""
platforms = [Platform("x86_64", "linux"; libc="glibc", cxxstring_abi="cxx11")]
platforms = expand_cxxstring_abis(platforms)
products = [FileProduct("amdgcn/bitcode/", :bitcode_path)]
dependencies = [
#BuildDependency(PackageSpec(; name="LLVM_full_jll", version=v"12.0.1")),
BuildDependency(PackageSpec(; name="ROCmLLVM_jll", version=v"4.2.0")),
Dependency("Zlib_jll"),
]
build_tarballs(
ARGS, name, version, sources, script, platforms, products, dependencies;
preferred_gcc_version=v"9", preferred_llvm_version=v"12")
```
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [July 8, 2022, 7:01pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/2 "2022-07-08T19:01:29Z")
Well, if you want to use your own clang (why?) you have to manually pass to the your compiler all flags we pass to the compiler wrapper, they are a few:
```bash
% julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("x86_64", "linux"))'
sandbox:${WORKSPACE} # cat $(which clang)
#!/bin/bash
# This compiler wrapper script brought into existence by `generate_compiler_wrappers!()`
if ["x${SUPER_VERBOSE}" = "x"]; then
vrun() { "$@"; }
else
vrun() { echo -e "\e[96m$@\e[0m" >&2; "$@"; }
fi
ARGS=( "$@" )
PRE_FLAGS=()
POST_FLAGS=()
PRE_FLAGS+=( -target x86_64-linux-gnu )
PRE_FLAGS+=( --sysroot=/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root )
PRE_FLAGS+=( --gcc-toolchain=/opt/x86_64-linux-gnu )
PRE_FLAGS+=( -rtlib=libgcc )
PRE_FLAGS+=( -stdlib=libstdc++ )
if [[" ${ARGS[@]} " != *' -x assembler '* ]]; then
PRE_FLAGS+=( -march=x86-64 )
PRE_FLAGS+=( -mtune=generic )
fi
if [[" ${ARGS[@]} " != *' -c '* ]] && [[" ${ARGS[@]} " != *' -E '* ]] && [[" ${ARGS[@]} " != *' -M '* ]] && [[" ${ARGS[@]} " != *' -fsyntax-only '* ]]; then
POST_FLAGS+=( -fuse-ld=x86_64-linux-gnu )
fi
export LD_LIBRARY_PATH="/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib"
if [[" ${ARGS[@]} " == *"-march="* ]]; then
echo "BinaryBuilder: Cannot force an architecture via -march" >&2
exit 1
fi
if [${USE_CCACHE} == "true"]; then
CCACHE="ccache"
fi
vrun ${CCACHE} /opt/x86_64-linux-musl/bin/clang "${PRE_FLAGS[@]}" "${ARGS[@]}" "${POST_FLAGS[@]}"
```
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [July 8, 2022, 7:13pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/3 "2022-07-08T19:13:20Z")
We need/want to use the clang provided by ROCmLLVM\_jll so that our device libs bitcode is generated correctly (ROCm’s LLVM has many changes that differ from Julia’s LLVM).
---
### Author:  [@pxl-th](https://discourse.julialang.org/u/pxl-th)
#### Post date: [July 10, 2022, 1:43pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/4 "2022-07-10T13:43:54Z")
That worked, thanks!
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [July 10, 2022, 2:01pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/5 "2022-07-10T14:01:01Z")
I don’t know what you eventually did, but you could define your own shell wrapper like the one above, just changing the path `/opt/x86_64-linux-musl/bin/clang` in the last line (or edit directly `$(which clang)` 😱).
---
### Author:  [@pxl-th](https://discourse.julialang.org/u/pxl-th)
#### Post date: [July 10, 2022, 6:27pm UTC](https://discourse.julialang.org/t/rocm-device-libs-binarybuilder/83962/6 "2022-07-10T18:27:45Z")
Yes, I did use a shell wrapper.
I managed to build all the dependencies needed for the rocBLAS with minimal number of patches and disabled features, basically following any rocm build guide.
Here are the scripts in order: [GitHub - pxl-th/rocm-bb](https://github.com/pxl-th/rocm-bb)
However, I’m still getting segfault when dlopening rocBLAS, with the same error as in [[New Package] Add rocBLAS 4.2.0 by jpsamaroo · Pull Request #4255 · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/pull/4255) (jumping to `-1` address in `.init_array`).
Thus I’m not sure that the issue is due to musl & glibc mixing…
Here’s the segfault during BB audit:
```julia
[ Info: Beginning audit of /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/include/hip to point to ../../hip/include/hip
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/tools/clang to point to ../bin/clang
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/tools/lld to point to ../bin/lld
[ Info: Skipping binary analysis of hipcc
┌ Warning: librocblas.so.0.1 contains a `cpuid` instruction; refusing to analyze for minimum instruction set, as it may dynamically select the proper instruction set internally. Would have chosen avx2, instead choosing x86_64.
└ @ BinaryBuilder.Auditor ~/.julia/packages/BinaryBuilder/wohhx/src/auditor/instruction_set.jl:163
[ Info: /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/rocblas/lib/librocblas.so.0.1 locks us to cxx11
[Info: Checking rocblas/lib/librocblas.so.0.1 with RPath list SubString{String}["", "/workspace/destdir/hip/lib", "/workspace/destdir/lib"]
[ Info: Linked library libz.so.1 has been auto-mapped to libz.so.1
[ Info: Linked library libamdhip64.so.4 has been auto-mapped to libamdhip64.so.4
[ Info: Ignored system libraries libm.so.6, libpthread.so.0, librt.so.1, libstdc++.so.6, libgcc_s.so.1, libdl.so.2, libc.so.6, ld-linux-x86-64.so.2
[ Info: Checking shared library rocblas/lib/librocblas.so.0.1
signal (11): Segmentation fault
in expression starting at none:2
unknown function (ip: (nil))
Allocations: 2723 (Pool: 2714; Big: 9); GC: 0
```