Hello,
I’m working on enabling Julia to target GPUs via Polly. Polly needs the LLVM NVPTX backend to produce kernel code. This currently is failing because TargetRegistry::lookupTarget couldn’t find “nvptx64-nvidia-cuda”. How do I include the NVPTX backend functionality within libjulia.so ?
Thank You,
Sanjay
On Julia 0.6 master, it should already be enabled since the merge of https://github.com/JuliaLang/julia/pull/19323.
Does this apply when Julia’s using external LLVM builds, i.e. when LLVM_CONFIG is set in Make.user or Make.inc ?
In that case julia doesn’t build the external llvm so it’s entirely up to you to configure it.
Thanks for replying @yuyichao !
I’ve already targeted X86 and NVPTX in my LLVM build. But, I’m unable to find a symbol with NVPTX as a substring in the output of objdump -x usr/lib/libjulia.so .
Hi Singam,
I would check the following things:
Does the NVPTX backend work in your externally compiled LLVM. Please run llc on one of its test cases to see it actually outputs PTX
Can you post the linker command lines used to link libjulia.so? Do they include the LLVM PTX backend?
Best,
Tobias
Yes. When opt (from the LLVM build meant for Julia) was called on the IR meant for Polly (“-polly-dump-before”), it generated PTX kernel code and a .
There are 3 statements that indicate that the LLVM build must Target NVPTX too,
Since this file’s in < julia_src >/deps/, I’m assuming these options are used when Julia isn’t supplied with an external build. I couldn’t find a statement that uses these options to link Julia to NVPTX back-end when Julia’s supplied with an external LLVM build.
Can you just run “make VERBOSE=1” to see the output that is used to link with LLVM. What I am looking for are the following lines:
LLVMLINK += (shell (LLVM_CONFIG_HOST) --ldflags) (shell (LLVM_CONFIG_HOST) --libs (LLVM_LIBS)) (shell (LLVM_CONFIG_HOST) --ldflags) (shell (LLVM_CONFIG_HOST) --system-libs 2> /dev/null)
else
ifeq ((LLVM_USE_CMAKE),1)
LLVMLINK += (shell (LLVM_CONFIG_HOST) --ldflags) -lLLVM
else
ifeq ((OS),WINNT)
LLVMLINK += (shell (LLVM_CONFIG_HOST) --ldflags) -lLLVM-(LLVM_VER_SHORT)
else
LLVMLINK += (shell (LLVM_CONFIG_HOST) --ldflags) -lLLVM-(shell (LLVM_CONFIG_HOST) --version)
Did you try to set LLVM_USE_SHLIB? Maybe this helps?
You could also try to enable BUILD_SHARED_LIBS in the LLVM build. After this, ‘lld’ will show the links that e.g. ‘opt’ is linked to.
Best,
Tobias
Hello @Tobias_Grosser1,
I think you’re talking about these lines in src/Makefile.
Without the ifeq ($(USE_SYSTEM_LLVM),1) barrier, these lines execute irrespective of whether Julia’s linked to an external build of LLVM. The LLVM_CONFIG_HOST could point to the llvm-config in an LLVM build in < julia_src >/deps/.
Here’s the output after rebuilding with BUILD_SHARED_LIBS,
$ ldd bin/opt
linux-vdso.so.1 => (0x00007ffcfcb2f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2f016b6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2f014b1000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2f01288000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2f0106f000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2f00d6a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2f00a64000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2f0084e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2f00488000)
/lib64/ld-linux-x86-64.so.2 (0x000055b40d89b000)
And before the rebuild,
$ldd bin/opt
linux-vdso.so.1 => (0x00007fff4467a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f917e700000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f917e4fb000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f917e2d2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f917e0b9000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f917ddb4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f917daae000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f917d898000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f917d4d2000)
/lib64/ld-linux-x86-64.so.2 (0x0000561535b65000)
I don’t think anything has changed, which is surprising.
Here’s the linker command that creates libjulia-debug.so,
g++ -m64 -shared -pipe -fPIC -fno-rtti -pedantic -O0 -ggdb2 -DJL_DEBUG_BUILD -fstack-protector-all -D_GNU_SOURCE -I. -I/home/sanjay/Software/polly_julia/julia_src/src -I/home/sanjay/Software/polly_julia/julia_src/src/flisp -I/home/sanjay/Software/polly_julia/julia_src/src/support -I/home/sanjay/Software/polly_julia/julia_src/usr/include -I/home/sanjay/Software/polly_julia/julia_src/usr/include -DLIBRARY_EXPORTS -I/home/sanjay/Software/polly_julia/julia_src/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wpointer-arith -Wundef -DJL_BUILD_ARCH='"x86_64"' -DJL_BUILD_UNAME='"Linux"' -I/home/sanjay/Software/polly_julia/llvm_src/include -I/home/sanjay/Software/polly_julia/llvm_src/tools/polly/include -I/home/sanjay/Software/polly_julia/llvm_build/tools/polly/include -DUSE_POLLY -fopenmp "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys-debug.so\"" ./jltypes.dbg.obj ./gf.dbg.obj ./typemap.dbg.obj ./ast.dbg.obj ./builtins.dbg.obj ./module.dbg.obj ./interpreter.dbg.obj ./symbol.dbg.obj ./dlload.dbg.obj ./sys.dbg.obj ./init.dbg.obj ./task.dbg.obj ./array.dbg.obj ./dump.dbg.obj ./toplevel.dbg.obj ./jl_uv.dbg.obj ./datatype.dbg.obj ./simplevector.dbg.obj ./APInt-C.dbg.obj ./runtime_intrinsics.dbg.obj ./runtime_ccall.dbg.obj ./threadgroup.dbg.obj ./threading.dbg.obj ./stackwalk.dbg.obj ./gc.dbg.obj ./gc-debug.dbg.obj ./gc-pages.dbg.obj ./method.dbg.obj ./jlapi.dbg.obj ./signal-handling.dbg.obj ./safepoint.dbg.obj ./jloptions.dbg.obj ./timing.dbg.obj ./subtype.dbg.obj ./rtutils.dbg.obj ./codegen.dbg.obj ./jitlayers.dbg.obj ./disasm.dbg.obj ./debuginfo.dbg.obj ./llvm-simdloop.dbg.obj ./llvm-ptls.dbg.obj ./llvm-gcroot.dbg.obj ./cgmemmgr.dbg.obj -Wl,-rpath,'$ORIGIN' -Wl,-rpath,'$ORIGIN/julia' -Wl,-z,origin -o /home/sanjay/Software/polly_julia/julia_src/usr/lib/libjulia-debug.so.0.6.0 -Wl,-Bsymbolic-functions -Wl,--whole-archive ./flisp/libflisp-debug.a -Wl,--whole-archive ./support/libsupport-debug.a -L/home/sanjay/Software/polly_julia/julia_src/usr/lib -L/home/sanjay/Software/polly_julia/julia_src/usr/lib /home/sanjay/Software/polly_julia/julia_src/usr/lib/libuv.a /home/sanjay/Software/polly_julia/julia_src/usr/lib/libutf8proc.a -Wl,--no-whole-archive -lPolly -lPollyISL -lPollyPPCG -Wl,-rpath=/home/sanjay/Software/polly_julia/llvm_build/lib -lGPURuntime -L/home/sanjay/Software/polly_julia/llvm_build/lib -lLLVMLTO -lLLVMPasses -lLLVMObjCARCOpts -lLLVMSymbolize -lLLVMDebugInfoPDB -lLLVMDebugInfoDWARF -lLLVMCoverage -lLLVMTableGen -lLLVMLineEditor -lLLVMOrcJIT -lLLVMMIRParser -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMObjectYAML -lLLVMLibDriver -lLLVMOption -lgtest_main -lgtest -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMCodeGen -lLLVMTarget -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMScalarOpts -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMAnalysis -lLLVMObject -lLLVMMCParser -lLLVMMC -lLLVMBitReader -lLLVMProfileData -lLLVMCore -lLLVMSupport -lLLVMDemangle -L/home/sanjay/Software/polly_julia/llvm_build/lib -lrt -ldl -ltinfo -lpthread -lz -lm -Wl,--no-as-needed -ldl -lrt -lpthread -Wl,--export-dynamic,--as-needed,--no-whole-archive /home/sanjay/Software/polly_julia/julia_src/usr/lib/libunwind-generic.a /home/sanjay/Software/polly_julia/julia_src/usr/lib/libunwind.a -Wl,--version-script=/home/sanjay/Software/polly_julia/julia_src/src/julia.expmap -Wl,-soname=libjulia-debug.so.0.6
libjulia-debug.so is linked to NVPTX back-end through “-lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter” yet still doesn’t have any symbol that has “NVPTX” as a substring.
-Sanjay
Hello,
I was enabling BUILD_SHARED_LIBS in < llvm_src >/CMakeLists.txt and that didn’t reflect in < llvm_build >/CMakeCache.txt (which had BUILD_SHARED_LIBS:BOOL=OFF). Setting BUILD_SHARED_LIBS=ON through the cmake command solved the problem. According to ldd, opt had the following dependencies on the NVPTX back-end,
Here’s the list of dependencies of libjulia-debug.so according to ldd, which is clearly missing any libLLVMNVPTX*.so.
According to, [quote=“Tobias_Grosser1, post:10, topic:2756”]
http://www.bnikolic.co.uk/blog/gnu-ld-as-needed.html
[/quote]
and linker - Why does the order in which libraries are linked sometimes cause errors in GCC? - Stack Overflow, all those libraries which aren’t required by the .o files aren’t linked into the final executable or shared object. According to this logic,
Would manually specifying that PollyPPCG requires certain parts of the NVPTX back-end help ? For e.g. < llvm_src >/tools/opt/CMakeLists.txt explicitly mentions to link all of ${LLVM_TARGETS_TO_BUILD} to opt.
Thank You,
Sanjay Srivallabh
Hello,
I was enabling BUILD_SHARED_LIBS in < llvm_src >/CMakeLists.txt and that
didn’t reflect in < llvm_build >/CMakeCache.txt (which had
BUILD_SHARED_LIBS:BOOL=OFF). Setting BUILD_SHARED_LIBS=ON through the
cmake command solved the problem. According to ldd, opt had the following
dependencies on the NVPTX back-end,
- libLLVMNVPTXCodeGen.so.5
- libLLVMNVPTXDesc.so.5
- libLLVMNVPTXInfo.so.5
- libLLVMNVPTXAsmPrinter.so.5
Here’s the list of dependencies of libjulia-debug.so according to
ldd,
which is clearly missing any libLLVMNVPTX*.so.According to, [quote=“Tobias_Grosser1, post:10, topic:2756”]
The new --as-needed option to the GNU linker | B. Nikolic Software and Computing Blog
[/quote]and linker - Why does the order in which libraries are linked sometimes cause errors in GCC? - Stack Overflow, all those libraries which
aren’t required by the .o files aren’t linked into the final executable
or shared object. According to this logic,
- PollyISL and PollyPPCG aren’t chucked out since they’re required by
Polly, which is in turn required by one of the LLVM files.- NVPTX back-end’s not included since PollyPPCG has calls to it
indirectly via TargetRegistry::lookupTarget and
createTargetTransformInfoWrapperPass, which is already included via
LLVMTarget and LLVMAnalysis.Would manually specifying that PollyPPCG requires certain parts of the
NVPTX back-end help ? For e.g. < llvm_src >/tools/opt/CMakeLists.txt
explicitly mentions to link all of ${LLVM_TARGETS_TO_BUILD} to opt.
Could you disable the as-needed functionality to force the linker to
include all libraries?
Best,
Tobias
If this does not work, you could – as a workaround – also just try to call some of the NVPTX functions to make sure they are linked in. LLVM does the following in LinkAllPasses.h:
ForcePassLinking() {
// We must reference the passes in such a way that compilers will not
// delete it all as dead code, even with whole program optimization,
// yet is effectively a NO-OP. As the compiler isn't smart enough
// to know that getenv() never returns -1, this will do the job.
if (std::getenv("bar") != (char*) -1)
return;
(void) llvm::createAAEvalPass();
(void) llvm::createAggressiveDCEPass();
I don’t think LLVM recommends using BUILD_SHARED_LIBS=ON as that creates a separate shared library for every folder in LLVM. They have differently spelled flags (check deps/llvm.mk for exactly what we use) to link all of LLVM into a single shared library.
Hello @Tobias_Grosser1,
Adding “-Wl,–no-as-needed” included the libLLVMNVPTX*.so files and Polly’s GPURuntime also in libjulia-debug.so, so Julia doesn’t exit while dereferencing polly_getKernel.
The “nvptx64-nvidia-cuda” triple still isn’t recognized and kernel_gemm-after.ll (kernel_gemm’s the function sent to Polly) has the following lines,
@kernel_0 = private unnamed_addr constant [1 x i8] zeroinitializer
@kernel_0_name = private unnamed_addr constant [9 x i8] c"kernel_0\00"
kernel_gemm defaults to running on the CPU with nvprof not detecting any GPU kernels being launched.
@tkeiman: BUILD_SHARED_LIBS is certainly only a workaround to get this problem debuggged. We certainly need to get this running with a configuration that is commonly shipped to end users.
@singam-sanjay: Interesting. I am afraid you need to debug this. Maybe the GPU backends are not initialized / registered. Can you step through the code and see at which place LLVM realizes the the NVPTX backend is not available?
We’re using this to know which LLVM libraries opt and libjulia-debug.so link to.
I guess that’s the case. I’d need help with understanding how Julia calls LLVMInitializeX86Target and would apply the same to LLVMInitializeNVPTXTarget. It seems that < llvm_build >/lib/target/cmake_install.cmake takes care of registering all the enabled backends for the LLVM build.
There are certain patches that Julia applies to the LLVM source that are associated with enabling NVPTX backend. Since these aren’t being applied, is that causing a problem ?
It realizes it at TargetRegistry::lookupTarget. Does this mean that LLVMInitializeNVPTXTarget isn’t being called ?
I guess that’s the case. I’d need help with understanding how Julia calls
LLVMInitializeX86Target and would apply the same to
LLVMInitializeNVPTXTarget. It seems that < llvm_build/lib/target/cmake_install.cmake takes care of registering all the
enabled backends for the LLVM build.There are certain
patches
that Julia applies to the LLVM source that are associated with enabling
NVPTX backend. Since these aren’t being applied, is that causing a
problem ?It realizes it at
TargetRegistry::lookupTarget.
Does this mean that LLVMInitializeNVPTXTarget isn’t being called ?
Can you figure this out by adding a print statement?
Best,
Tobias
The NVPTX Backend was integrated with a few changes to src/codegen.cpp. Will send out a commit as soon as possible.