# Problem when loading Intel MKL shared library to use with C written shared library

**URL:** <https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323>\
**Category:** General Usage\
**Tags:** ccall, mkl\
**Created:** [October 6, 2021, 6:30pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323 "2021-10-06T18:30:19Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 6, 2021, 6:30pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/1 "2021-10-06T18:30:19Z")

</div>

Hi, I have written an optimized C library to do some matrix algebra basically, speeding up some einsum operations.  
I can compile and use the library from C and C++, it links to Intel MKL and is able to run normally. This is how I am trying to use if from Julia:

```julia
W = 1.0*Vector(1:4)
X = Vector([4:10])*1.0
v = Vector([1:6])
P = Vector([0;0])
XV = Vector(1:4)
using Libdl

Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_core.so")
Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so")
Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ccall((:face_split_product_dgemm_batched, "/opt/rh/devtoolset-9/root/usr/lib/libface_splitting_c.so"), Cvoid, (Ptr{Float64}, Ptr{Float64}, Ptr{Float64}, Int64, Int64, Int64, Ptr{Float64}, Ptr{Float64}), W, X, v, 2, 2, 3, P, XV)

```

I know that my written library `libface_splitting_c.so` links to Intel MKL and its battery of .so files. I am trying to `dlopen` one of those and this is what I am getting:

```julia
julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[2]:1

```

This is how I compile the libraries:

```julia
CPP=g++
CC=gcc

MKLROOT = /opt/intel/compilers_and_libraries_2020.4.304/linux/mkl

# include and link directories must have been amended by mklvsrs.sh script from intel mkl

CFLAGS=-c -I. -fopenmp -m64 -Wl,--no-as-needed -O3
LFLAGS=-L. -fopenmp -m64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lpthread -lm -ldl
SHFLAGS= -fpic
DEPS = face_splitting_product.cpp face_splitting_product.hpp face_splitting_product.c face_splitting_product.h

all: face_splitting_product.o face_splitting_product_c.o face_splitting.so face_splitting_c.so

face_splitting_product.o: face_splitting_product.cpp $(DEPS)
	$(CPP) face_splitting_product.cpp -o face_splitting_product.o $(CFLAGS) $(SHFLAGS)
	
face_splitting_product_c.o: face_splitting_product.c $(DEPS)
	$(CC) face_splitting_product.c -o face_splitting_product_c.o $(CFLAGS) $(SHFLAGS)

face_splitting.so: face_splitting_product.o
	$(CPP) -shared face_splitting_product.o -o libface_splitting.so $(LFLAGS)

face_splitting_c.so: face_splitting_product_c.o
	$(CC) -shared face_splitting_product_c.o -o libface_splitting_c.so $(LFLAGS)

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 6, 2021, 9:48pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/2 "2021-10-06T21:48:55Z")

</div>

I’m not at the computer, which MKL library defines the symbol `mkl_lapack_zhesvxx`?

---

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 7, 2021, 6:06pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/3 "2021-10-07T18:06:38Z")

</div>

> [@Roman\_Istomin](#):
>
> `mkl_lapack_zhesvxx`

doesn’t look like it is defined anywhere 🙂

```julia
[root@c5981136386c FaceSplitting.cpp] grep -Ir 'mkl_lapack_zhesvxx' /opt/
[root@c5981136386c FaceSplitting.cpp] 

```

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 7, 2021, 6:20pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/4 "2021-10-07T18:20:02Z")

</div>

Not specific to your problem, but sharing a link to MKL.jl for future readers: [GitHub - JuliaLinearAlgebra/MKL.jl: Intel MKL linear algebra backend for Julia](https://github.com/JuliaLinearAlgebra/MKL.jl)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 7, 2021, 6:21pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/5 "2021-10-07T18:21:55Z")

</div>

Can you please do something like

```julia
for lib in /dir/where/MKL/libraries/are; do
    echo "Library: ${lib}"
    nm "${lib}" | grep mkl_lapack_zhesvxx
end

```

You’ll find libraries that both define and reference (without defining) `mkl_lapack_zhesvxx`. The latter will appear like

```julia
xxxxx U mkl_lapack_zhesvxx

```

where the `U` stands for “undefined”, we’re interested in the others.

---

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 7, 2021, 6:46pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/6 "2021-10-07T18:46:20Z")

</div>

this is what I am getting with slight modification on the command (done at the end instead of end)

```julia
[root@c5981136386c FaceSplitting.cpp]# for lib in /opt/intel/mkl/lib/intel64/*; do echo "Library: ${lib}"; nm "${lib}" | grep mkl_lapack_zhesvxx; done
Library: /opt/intel/mkl/lib/intel64/libmkl_avx2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx512_mic.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx512.so
Library: /opt/intel/mkl/lib/intel64/libmkl_avx.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blacs_sgimpt_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_blas95_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_blas95_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_cdft_core.a
Library: /opt/intel/mkl/lib/intel64/libmkl_cdft_core.so
Library: /opt/intel/mkl/lib/intel64/libmkl_core.a
0000000000000000 T mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_core.so
0000000000c4c830 T mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_def.so
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_ilp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_ilp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.a
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so
                 U mkl_lapack_zhesvxx
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_lapack95_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_lapack95_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_mc3.so
Library: /opt/intel/mkl/lib/intel64/libmkl_mc.so
Library: /opt/intel/mkl/lib/intel64/libmkl_pgi_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_pgi_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_rt.so
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_lp64.a
Library: /opt/intel/mkl/lib/intel64/libmkl_scalapack_lp64.so
Library: /opt/intel/mkl/lib/intel64/libmkl_sequential.a
Library: /opt/intel/mkl/lib/intel64/libmkl_sequential.so
Library: /opt/intel/mkl/lib/intel64/libmkl_tbb_thread.a
Library: /opt/intel/mkl/lib/intel64/libmkl_tbb_thread.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx512_mic.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx512.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_avx.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_cmpt.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_def.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc2.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc3.so
Library: /opt/intel/mkl/lib/intel64/libmkl_vml_mc.so
Library: /opt/intel/mkl/lib/intel64/locale
nm: Warning: '/opt/intel/mkl/lib/intel64/locale' is a directory

```

---

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 7, 2021, 6:51pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/7 "2021-10-07T18:51:14Z")

</div>

so, it looks like it is defined here: `Library: /opt/intel/mkl/lib/intel64/libmkl_core.so `, I am adding that library and then again try adding libmkl\_intel\_lp64.so:

```julia
julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_core.so")
Ptr{Nothing} @0x0000000002c08a90

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[21]:1

```

---

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 7, 2021, 7:13pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/8 "2021-10-07T19:13:22Z")

</div>

what I decided to do is to try and add all of the linraries to Julia and see if that resolves the problem, apparently, it does not

```julia
julia> for file in libs
           if (occursin(".so", file))
               try 
                   Libdl.dlopen("/opt/intel/mkl/lib/intel64/"*file)
                   println("added file $file")
               catch e
                   println("cannot open file $file")
               end
           end
       end
cannot open file libmkl_avx.so
cannot open file libmkl_avx2.so
cannot open file libmkl_avx512.so
cannot open file libmkl_avx512_mic.so
added file libmkl_blacs_intelmpi_ilp64.so
added file libmkl_blacs_intelmpi_lp64.so
cannot open file libmkl_blacs_openmpi_ilp64.so
cannot open file libmkl_blacs_openmpi_lp64.so
cannot open file libmkl_blacs_sgimpt_ilp64.so
cannot open file libmkl_blacs_sgimpt_lp64.so
cannot open file libmkl_cdft_core.so
added file libmkl_core.so
cannot open file libmkl_def.so
cannot open file libmkl_gf_ilp64.so
cannot open file libmkl_gf_lp64.so
cannot open file libmkl_gnu_thread.so
cannot open file libmkl_intel_ilp64.so
cannot open file libmkl_intel_lp64.so
cannot open file libmkl_intel_thread.so
cannot open file libmkl_mc.so
cannot open file libmkl_mc3.so
cannot open file libmkl_pgi_thread.so
added file libmkl_rt.so
cannot open file libmkl_scalapack_ilp64.so
cannot open file libmkl_scalapack_lp64.so
cannot open file libmkl_sequential.so
cannot open file libmkl_tbb_thread.so
cannot open file libmkl_vml_avx.so
cannot open file libmkl_vml_avx2.so
cannot open file libmkl_vml_avx512.so
cannot open file libmkl_vml_avx512_mic.so
cannot open file libmkl_vml_cmpt.so
cannot open file libmkl_vml_def.so
cannot open file libmkl_vml_mc.so
cannot open file libmkl_vml_mc2.so
cannot open file libmkl_vml_mc3.so

julia> Libdl.dlopen("/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so")
ERROR: could not load library "/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so"
/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so: undefined symbol: mkl_lapack_zhesvxx
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] dlopen (repeats 2 times)
   @ ./libdl.jl:114 [inlined]
 [3] top-level scope
   @ REPL[48]:100: 

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 12, 2021, 12:00pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/9 "2021-10-12T12:00:16Z")

</div>

It looks like you need to dlopen the libraries with `RTLD_GLOBAL`:

```julia
julia> using Libdl

julia> libdir = "/home/mose/.julia/artifacts/72d4adc3ef9236a92f4fefeb0291cb6e8aaae2d7/lib"
"/home/mose/.julia/artifacts/72d4adc3ef9236a92f4fefeb0291cb6e8aaae2d7/lib"

julia> flags = RTLD_LAZY|RTLD_DEEPBIND|RTLD_GLOBAL
0x00000046

julia> dlopen(joinpath(libdir, "libmkl_core.so"), flags)
Ptr{Nothing} @0x00000000026a1980

julia> dlopen(joinpath(libdir, "libmkl_gnu_thread.so"), flags)
Ptr{Nothing} @0x0000000002dbc850

julia> dlopen(joinpath(libdir, "libmkl_intel_lp64.so"), flags)
Ptr{Nothing} @0x0000000002f6daf0

```

---

<div class="post-metadata">

**Author:** ![Roman\_Istomin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roman_istomin/32/29750_2.png) [@Roman\_Istomin](https://discourse.julialang.org/u/Roman_Istomin)\
**Post date:** [October 12, 2021, 3:13pm UTC](https://discourse.julialang.org/t/problem-when-loading-intel-mkl-shared-library-to-use-with-c-written-shared-library/69323/10 "2021-10-12T15:13:36Z")

</div>

@giordano That helped to load the libraries, thank you very much. Now I am seeing another problem that the actual `ccal` crashes the REPL, but that is another topic, outside this thread