Problem when loading Intel MKL shared library to use with C written shared library

General Usage
,
1

Hi, I have written an optimized C library to do some matrix algebra basically, speeding up some einsum operations.
I can compile and use the library from C and C++, it links to Intel MKL and is able to run normally. This is how I am trying to use if from Julia:

W = 1.0*Vector(1:4)
X = Vector([4:10])*1.0
v = Vector([1:6])
P = Vector([0;0])
XV = Vector(1:4)
using Libdl

Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_core.so")
Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so")
Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ccall((:face_split_product_dgemm_batched, "/opt/rh/devtoolset-9/root/usr/lib/libface_splitting_c.so"), Cvoid, (Ptr{Float64}, Ptr{Float64}, Ptr{Float64}, Int64, Int64, Int64, Ptr{Float64}, Ptr{Float64}), W, X, v, 2, 2, 3, P, XV)

I know that my written library libface_splitting_c.so links to Intel MKL and its battery of .so files. I am trying to dlopen one of those and this is what I am getting:

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[2]:1

This is how I compile the libraries:

CPP=g++
CC=gcc

MKLROOT = /opt/intel/compilers_and_libraries_2020.4.304/linux/mkl

# include and link directories must have been amended by mklvsrs.sh script from intel mkl

CFLAGS=-c -I. -fopenmp -m64 -Wl,--no-as-needed -O3
LFLAGS=-L. -fopenmp -m64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lpthread -lm -ldl
SHFLAGS= -fpic
DEPS = face_splitting_product.cpp face_splitting_product.hpp face_splitting_product.c face_splitting_product.h

all: face_splitting_product.o face_splitting_product_c.o face_splitting.so face_splitting_c.so

face_splitting_product.o: face_splitting_product.cpp $(DEPS)
	$(CPP) face_splitting_product.cpp -o face_splitting_product.o $(CFLAGS) $(SHFLAGS)
	
face_splitting_product_c.o: face_splitting_product.c $(DEPS)
	$(CC) face_splitting_product.c -o face_splitting_product_c.o $(CFLAGS) $(SHFLAGS)

face_splitting.so: face_splitting_product.o
	$(CPP) -shared face_splitting_product.o -o libface_splitting.so $(LFLAGS)

face_splitting_c.so: face_splitting_product_c.o
	$(CC) -shared face_splitting_product_c.o -o libface_splitting_c.so $(LFLAGS)
2

I’m not at the computer, which MKL library defines the symbol mkl_lapack_zhesvxx?

3

doesn’t look like it is defined anywhere :slight_smile:

[root@c5981136386c FaceSplitting.cpp] grep -Ir 'mkl_lapack_zhesvxx' /opt/
[root@c5981136386c FaceSplitting.cpp]
4

Not specific to your problem, but sharing a link to MKL.jl for future readers: GitHub - JuliaLinearAlgebra/MKL.jl: Intel MKL linear algebra backend for Julia

5

Can you please do something like

for lib in /dir/where/MKL/libraries/are; do
    echo "Library: ${lib}"
    nm "${lib}" | grep mkl_lapack_zhesvxx
end

You’ll find libraries that both define and reference (without defining) mkl_lapack_zhesvxx. The latter will appear like

xxxxx U mkl_lapack_zhesvxx

where the U stands for “undefined”, we’re interested in the others.

6

this is what I am getting with slight modification on the command (done at the end instead of end)

[root@c5981136386c FaceSplitting.cpp]# for lib in /opt/intel/mkl/lib/intel64/*; do     echo "Library: ${lib}";     nm "${lib}" | grep mkl_lapack_zhesvxx; done
Library: /opt/intel/mkl/lib/intel64/libmkl_avx2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx512_mic.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx512.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blas95_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blas95_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_cdft_core.a
Library: /opt/intel/mkl/lib/intel64/libmkl_cdft_core.so
Library: /opt/intel/mkl/lib/intel64/libmkl_core.a
0000000000000000 T mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_core.so
0000000000c4c830 T mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_def.so
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_ilp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_ilp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_lapack95_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_lapack95_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_mc3.so
Library: /opt/intel/mkl/lib/intel64/libmkl_mc.so
Library: /opt/intel/mkl/lib/intel64/libmkl_pgi_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_pgi_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_rt.so
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_sequential.a
Library: /opt/intel/mkl/lib/intel64/libmkl_sequential.so
Library: /opt/intel/mkl/lib/intel64/libmkl_tbb_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_tbb_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx512_mic.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx512.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_cmpt.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_def.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc3.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc.so
Library: /opt/intel/mkl/lib/intel64/locale
nm: Warning: '/opt/intel/mkl/lib/intel64/locale' is a directory
7

so, it looks like it is defined here: Library: /opt/intel/mkl/lib/intel64/libmkl_core.so , I am adding that library and then again try adding libmkl_intel_lp64.so:

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_core.so")
Ptr{Nothing} @0x0000000002c08a90

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[21]:1
8

what I decided to do is to try and add all of the linraries to Julia and see if that resolves the problem, apparently, it does not

julia> for file in libs
           if (occursin(".so", file))
               try 
                   Libdl.dlopen("/opt/intel/mkl/lib/intel64/"*file)
                   println("added file $file")
               catch e
                   println("cannot open file $file")
               end
           end
       end
cannot open file libmkl_avx.so
cannot open file libmkl_avx2.so
cannot open file libmkl_avx512.so
cannot open file libmkl_avx512_mic.so
added file libmkl_blacs_intelmpi_ilp64.so
added file libmkl_blacs_intelmpi_lp64.so
cannot open file libmkl_blacs_openmpi_ilp64.so
cannot open file libmkl_blacs_openmpi_lp64.so
cannot open file libmkl_blacs_sgimpt_ilp64.so
cannot open file libmkl_blacs_sgimpt_lp64.so
cannot open file libmkl_cdft_core.so
added file libmkl_core.so
cannot open file libmkl_def.so
cannot open file libmkl_gf_ilp64.so
cannot open file libmkl_gf_lp64.so
cannot open file libmkl_gnu_thread.so
cannot open file libmkl_intel_ilp64.so
cannot open file libmkl_intel_lp64.so
cannot open file libmkl_intel_thread.so
cannot open file libmkl_mc.so
cannot open file libmkl_mc3.so
cannot open file libmkl_pgi_thread.so
added file libmkl_rt.so
cannot open file libmkl_scalapack_ilp64.so
cannot open file libmkl_scalapack_lp64.so
cannot open file libmkl_sequential.so
cannot open file libmkl_tbb_thread.so
cannot open file libmkl_vml_avx.so
cannot open file libmkl_vml_avx2.so
cannot open file libmkl_vml_avx512.so
cannot open file libmkl_vml_avx512_mic.so
cannot open file libmkl_vml_cmpt.so
cannot open file libmkl_vml_def.so
cannot open file libmkl_vml_mc.so
cannot open file libmkl_vml_mc2.so
cannot open file libmkl_vml_mc3.so

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[48]:100:
9

It looks like you need to dlopen the libraries with RTLD_GLOBAL:

julia> using Libdl

julia> libdir = "/home/mose/.julia/artifacts/72d4adc3ef9236a92f4fefeb0291cb6e8aaae2d7/lib"
"/home/mose/.julia/artifacts/72d4adc3ef9236a92f4fefeb0291cb6e8aaae2d7/lib"

julia> flags = RTLD_LAZY|RTLD_DEEPBIND|RTLD_GLOBAL
0x00000046

julia> dlopen(joinpath(libdir, "libmkl_core.so"), flags)
Ptr{Nothing} @0x00000000026a1980

julia> dlopen(joinpath(libdir, "libmkl_gnu_thread.so"), flags)
Ptr{Nothing} @0x0000000002dbc850

julia> dlopen(joinpath(libdir, "libmkl_intel_lp64.so"), flags)
Ptr{Nothing} @0x0000000002f6daf0
1 Like
10

@giordano That helped to load the libraries, thank you very much. Now I am seeing another problem that the actual ccal crashes the REPL, but that is another topic, outside this thread