ROCm-Device-Libs BinaryBuilder

Tooling
, ,
1

I’m having trouble building ROCm-Device-Libraries using BinaryBuilder with ROCmLLVM.
I’m trying to follow build recipe from the rocm-arch repository.
There they use ROCmLLVM to build it. But when specifying it in the BinaryBuilder I get errors I’m not sure show to fix.

The gist of the error:

The C compiler
  "/workspace/destdir/tools/clang"
is not able to compile a simple test program.
It fails with the following output:
  ...
  /workspace/destdir/tools/clang CMakeFiles/cmTC_7b11b.dir/testCCompiler.c.o -o cmTC_7b11b 
  /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crt1.o: No such file or directory
  /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crti.o: No such file or directory
  /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find crtbegin.o: No such file or directory
  /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find -lgcc
  /opt/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld: cannot find -lgcc_s
  clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

I’ve tried specifying toolchain file ${CMAKE_TARGET_TOOLCHAIN} as well as clang specific ${CMAKE_TARGET_TOOLCHAIN%.*}_clang.cmake.
But they overwrite the compiler with their own /opt/bin/x86_64-linux-gnu-libgfortran5-cxx11/x86_64-linux-gnu-clang

Am I missing somethings that needs to be configured when using different compiler?
I want to compile all the necessary packages to be able to build rocBLAS while reducing number of patches that disable features and potentially avoid mixing musl and glibc in the process.

Here’s the recipe:

using Pkg
using BinaryBuilder

name = "ROCmDeviceLibs"
version = v"4.2.0"

sources = [
    ArchiveSource(
        "https://github.com/RadeonOpenCompute/ROCm-Device-Libs/archive/rocm-$(version).tar.gz",
        "34a2ac39b9bb7cfa8175cbab05d30e7f3c06aaffce99eed5f79c616d0f910f5f"),
]

script = raw"""
cd ${WORKSPACE}/srcdir/ROCm-Device-Libs*/
mkdir build && cd build

ln -s ${prefix}/bin/clang ${prefix}/tools/clang
ln -s ${prefix}/bin/lld ${prefix}/tools/lld
export PATH="${prefix}/bin:${prefix}/tools:${PATH}"

CC=${prefix}/tools/clang \
CXX=${prefix}/tools/clang++ \
cmake \
    -DCMAKE_PREFIX_PATH=${prefix} \
    -DCMAKE_INSTALL_PREFIX=${prefix} \
    -DLLVM_DIR="${prefix}/lib/cmake/llvm" \
    -DClang_DIR="${prefix}/lib/cmake/clang" \
    ..
make -j${nproc}
make install
"""

platforms = [Platform("x86_64", "linux"; libc="glibc", cxxstring_abi="cxx11")]
platforms = expand_cxxstring_abis(platforms)
products = [FileProduct("amdgcn/bitcode/", :bitcode_path)]

dependencies = [
    #BuildDependency(PackageSpec(; name="LLVM_full_jll", version=v"12.0.1")),
    BuildDependency(PackageSpec(; name="ROCmLLVM_jll", version=v"4.2.0")),
    Dependency("Zlib_jll"),
]
build_tarballs(
    ARGS, name, version, sources, script, platforms, products, dependencies;
    preferred_gcc_version=v"9", preferred_llvm_version=v"12")
1 Like
2

Well, if you want to use your own clang (why?) you have to manually pass to the your compiler all flags we pass to the compiler wrapper, they are a few:

% julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("x86_64", "linux"))'
sandbox:${WORKSPACE} # cat $(which clang)
#!/bin/bash
# This compiler wrapper script brought into existence by `generate_compiler_wrappers!()`

if [ "x${SUPER_VERBOSE}" = "x" ]; then
    vrun() { "$@"; }
else
    vrun() { echo -e "\e[96m$@\e[0m" >&2; "$@"; }
fi

ARGS=( "$@" )
PRE_FLAGS=()
POST_FLAGS=()

PRE_FLAGS+=( -target x86_64-linux-gnu )
PRE_FLAGS+=( --sysroot=/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root )
PRE_FLAGS+=( --gcc-toolchain=/opt/x86_64-linux-gnu )
PRE_FLAGS+=( -rtlib=libgcc )
PRE_FLAGS+=( -stdlib=libstdc++ )


if [[ " ${ARGS[@]} " != *' -x assembler '* ]]; then
    PRE_FLAGS+=( -march=x86-64 )
    PRE_FLAGS+=( -mtune=generic )
fi


if [[ " ${ARGS[@]} " != *' -c '* ]] && [[ " ${ARGS[@]} " != *' -E '* ]] && [[ " ${ARGS[@]} " != *' -M '* ]] && [[ " ${ARGS[@]} " != *' -fsyntax-only '* ]]; then
    POST_FLAGS+=( -fuse-ld=x86_64-linux-gnu )
fi



export LD_LIBRARY_PATH="/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib"
if [[ " ${ARGS[@]} " == *"-march="* ]]; then
    echo "BinaryBuilder: Cannot force an architecture via -march" >&2
    exit 1
fi

if [ ${USE_CCACHE} == "true" ]; then
    CCACHE="ccache"
fi
vrun ${CCACHE} /opt/x86_64-linux-musl/bin/clang "${PRE_FLAGS[@]}" "${ARGS[@]}" "${POST_FLAGS[@]}"
3

We need/want to use the clang provided by ROCmLLVM_jll so that our device libs bitcode is generated correctly (ROCm’s LLVM has many changes that differ from Julia’s LLVM).

1 Like
4

That worked, thanks!

5

I don’t know what you eventually did, but you could define your own shell wrapper like the one above, just changing the path /opt/x86_64-linux-musl/bin/clang in the last line (or edit directly $(which clang) :scream:).

6

Yes, I did use a shell wrapper.
I managed to build all the dependencies needed for the rocBLAS with minimal number of patches and disabled features, basically following any rocm build guide.
Here are the scripts in order: GitHub - pxl-th/rocm-bb

However, I’m still getting segfault when dlopening rocBLAS, with the same error as in [New Package] Add rocBLAS 4.2.0 by jpsamaroo · Pull Request #4255 · JuliaPackaging/Yggdrasil · GitHub (jumping to -1 address in .init_array).
Thus I’m not sure that the issue is due to musl & glibc mixing…

Here’s the segfault during BB audit:

[ Info: Beginning audit of /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/lib/include/hip to point to ../../hip/include/hip
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/tools/clang to point to ../bin/clang
[ Info: Translating /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/tools/lld to point to ../bin/lld
[ Info: Skipping binary analysis of hipcc
┌ Warning: librocblas.so.0.1 contains a `cpuid` instruction; refusing to analyze for minimum instruction set, as it may dynamically select the proper instruction set internally.  Would have chosen avx2, instead choosing x86_64.
└ @ BinaryBuilder.Auditor ~/.julia/packages/BinaryBuilder/wohhx/src/auditor/instruction_set.jl:163
[ Info: /home/pxl-th/code/rocm-bb/build/x86_64-linux-gnu-cxx11/VcZuBGps/x86_64-linux-gnu-libgfortran5-cxx11/destdir/rocblas/lib/librocblas.so.0.1 locks us to cxx11
[ Info: Checking rocblas/lib/librocblas.so.0.1 with RPath list SubString{String}["", "/workspace/destdir/hip/lib", "/workspace/destdir/lib"]
[ Info: Linked library libz.so.1 has been auto-mapped to libz.so.1
[ Info: Linked library libamdhip64.so.4 has been auto-mapped to libamdhip64.so.4
[ Info: Ignored system libraries libm.so.6, libpthread.so.0, librt.so.1, libstdc++.so.6, libgcc_s.so.1, libdl.so.2, libc.so.6, ld-linux-x86-64.so.2
[ Info: Checking shared library rocblas/lib/librocblas.so.0.1

signal (11): Segmentation fault
in expression starting at none:2
unknown function (ip: (nil))
Allocations: 2723 (Pool: 2714; Big: 9); GC: 0