# Computing eigenvalues/eigenvectors using GPU?

**URL:** <https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396>\
**Category:** GPU\
**Created:** [July 15, 2018, 7:17pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396 "2018-07-15T19:17:03Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [July 15, 2018, 7:17pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/1 "2018-07-15T19:17:03Z")

</div>

Hi everyone

I have a Vega RX 64 and I would like to speed up eigenvalues decomposition on it.  
Does anybody know a (simple) solution with Julia ?

---

<div class="post-metadata">

**Author:** ![Ralph\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ralph_smith/32/10344_2.png) [@Ralph\_Smith](https://discourse.julialang.org/u/Ralph_Smith)\
**Post date:** [July 15, 2018, 10:37pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/2 "2018-07-15T22:37:18Z")

</div>

Short answer: AFAICT there isn’t even a not-simple solution.

To use a GPU effectively for dense eigensystem computations is quite difficult. The functionality was only recently added to the CUDA libraries (useless to you) and is conspicuously absent from the prominent projects using OpenCL (e.g. ArrayFire). There was some work in the clMagma project, but that seems to have been abandoned well before recent upgrades to the AMD software stack.

You say “decomposition”, so I assume you mean for dense matrices. For a few eigenvalues/vectors of sparse matrices there is more hope, but it’s still not simple.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [July 16, 2018, 5:28am UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/3 "2018-07-16T05:28:10Z")

</div>

> **[eigen-upstream/test at develop-upstream · rocmarchive/eigen-upstream](https://github.com/rocmarchive/eigen-upstream/tree/develop-upstream/test)**
>
> develop-upstream/test

(Look at the develop-upstream branch.)

They have unit tests for HIP, but I’m not sure what functionality is currently supported. The last PRs were very recent. HIP is cross platform, compiling through hcc for AMD, and CUDA for NVidia. There’re also pure CUDA unit tests in eigen.

So it may be a matter of creating a few shared libraries and `ccall`ing them.

Gregory Stoner of the ROCm project said months ago that there was some work underway for a ROCm-native in Julia, like CUDAnative. Not sure if there’s any progress on that front.

EDIT: HIP/ROCm only work on Linux. I think any distribution is supposed to work starting with Kernel 4.18, but at the moment only Ubuntu 16.04, and RHEL/CentOS 7.4 & 7.5 are officially supported.  
This probably doesn’t qualify as “simple”.

Also, want to point out that HIP/ROCm is pretty good. Their BLAS libraries were almost twice as fast on Vega as CLBLAS when I benchmarked them about five months ago.

---

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [July 18, 2018, 1:26pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/4 "2018-07-18T13:26:44Z")

</div>

I’m trying to use ArrayFire.jl, but when I try some linear algebra operations I get error with MKL

```julia
using ArrayFire;
a = rand(10, 10);
ad = AFArray(a);
det(ad)

```

`Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.`

Simple operations with variable `ad` like `sin(ad)` or `maximum(ad)` do work. However, operations like `det`, `qr` or `svd` don’t.

I’ve run the mkl script to fix PATH before running Julia

`source /opt/intel/mkl/bin/mklvars.sh intel64 `

And also, I followed the comments on web [1]:

```julia
conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

```

But none of this commands solved my problem.  
I’m open to suggestions 🙏

For reference:  
[1] [python - Intel MKL FATAL ERROR: Cannot load libmkl\_avx2.so or libmkl\_def.so - Stack Overflow](https://stackoverflow.com/questions/36659453/intel-mkl-fatal-error-cannot-load-libmkl-avx2-so-or-libmkl-def-so)

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [July 18, 2018, 1:56pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/5 "2018-07-18T13:56:18Z")

</div>

Conda is a Python package manager.

The following shared libraries (and versions) come with (a recent) ArrayFire:

```julia
libafcpu.so libaf.so libforge.so.1.1.0 libmkl_mc3.so
libafcpu.so.3 libaf.so.3 libglbinding.so.2 libmkl_mc.so
libafcpu.so.3.6.1 libaf.so.3.6.1 libglbinding.so.2.1.4 libmkl_tbb_thread.so
libafcuda.so libcublas.so.9.2 libmkl_avx2.so libnvrtc-builtins.so
libafcuda.so.3 libcufft.so.9.2 libmkl_avx512.so libnvrtc.so.9.2
libafcuda.so.3.6.1 libcusolver.so.9.2 libmkl_avx.so libOpenCL.so.1
libafopencl.so libcusparse.so.9.2 libmkl_core.so libtbb.so
libafopencl.so.3 libforge.so libmkl_def.so libtbb.so.2
libafopencl.so.3.6.1 libforge.so.1 libmkl_intel_lp64.so

```

You can see that `libmkl_avx2.so` and `libmkl_def.so` are in that list. Make sure they’re in your path, so Julia can find them.

However, that it was trying to use `mkl` sounds like it was trying to do the calculations on your CPU, and not your GPU?  
I’m not familiar with ArrayFire.

---

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [July 18, 2018, 2:01pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/6 "2018-07-18T14:01:39Z")

</div>

Yep, all these files are there. And in my PATH

```julia
> echo $LD_LIBRARY_PATH
:/opt/arrayfire/lib

```

Should I change others PATH variables too ?

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [July 18, 2018, 2:05pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/7 "2018-07-18T14:05:50Z")

</div>

Hmm, just installed ArrayFire and…

```julia
julia> a = rand(Float32, 10^3, 10^3);

julia> ad = AFArray(rand(Float32, 10^3, 10^3));

julia> @time det(ad)
Intel MKL FATAL ERROR: Cannot load libmkl_def.so.

```

EDIT:  
Also, when I tested last November, ArrayFire did not support Vega yet:  
[https://groups.google.com/forum/#!searchin/arrayfire-users/Vega|sort:date/arrayfire-users/SupCI8sTcdM/oiDH6bXtCAAJ](https://groups.google.com/forum/#!searchin/arrayfire-users/Vega%7Csort:date/arrayfire-users/SupCI8sTcdM/oiDH6bXtCAAJ)  
Not sure if that’s changed. Matmul works, but for large matrices is about 3x slower than HIP BLAS (still faster than CPU).

I am not sure what the problem is.  
[https://github.com/JuliaComputing/ArrayFire.jl/blob/943405b8b148d95e23ae9cea63c2579568f97cb0/src/wrap.jl](https://github.com/JuliaComputing/ArrayFire.jl/blob/943405b8b148d95e23ae9cea63c2579568f97cb0/src/wrap.jl)  
There are a lot of calls here, and most work.

I suspect it is the linear algebra ones that do not. They’re the ones that had problems then, too.

```julia
julia> qr(afc)
Intel MKL FATAL ERROR: Cannot load libmkl_def.so.

```

---

<div class="post-metadata">

**Author:** ![Ferran\_Mazzanti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ferran_mazzanti/32/7458_2.png) [@Ferran\_Mazzanti](https://discourse.julialang.org/u/Ferran_Mazzanti)\
**Post date:** [August 1, 2018, 8:32am UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/8 "2018-08-01T08:32:31Z")

</div>

Just for the record, I’ve just tried a fresh, new install of ArrayFire and I’m getting the exact same problems  
Intel MKL FATAL ERROR: Cannot load libmkl\_avx2.so or libmkl\_def.so.

…which makes it a pity as I was hoping to use the library for something more interesting that simply adding or multiplying matrices ☹

Does anybody know the status of that?

BTW I’ve installed in ubuntu mate 18.04

Best,

Ferran.

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [August 1, 2018, 2:52pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/9 "2018-08-01T14:52:33Z")

</div>

Hi, it works on osx for me.

```julia

julia> using ArrayFire
ArrayFire v3.6.0 (OpenCL, 64-bit Mac OSX, build 49a5436)
[0] APPLE: AMD Radeon Pro 560 Compute Engine, 4096 MB
-1- APPLE: Intel(R) HD Graphics 630, 1536 MB

julia> a = rand(Float32, 10^3, 10^3);

julia> ad = AFArray(rand(Float32, 10^3, 10^3));

julia> @time det(ad)
  1.808117 seconds (1.56 k allocations: 84.891 KiB)
(Inf, 0.0)

julia> @time det(ad)
  0.061030 seconds (7 allocations: 224 bytes)
(Inf, 0.0)

julia> @which det(ad)
det(_in::ArrayFire.AFArray) in ArrayFire at .julia/v0.6/ArrayFire/src/wrap.jl:1585

```

---

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [September 11, 2018, 6:14pm UTC](https://discourse.julialang.org/t/computing-eigenvalues-eigenvectors-using-gpu/12396/10 "2018-09-11T18:14:29Z")

</div>

For all of those that had the same issue, I’ve found the solution:

In summary, we need to load some paths with `LD_PRELOAD` . In my computer, `libmkl_avx2.so` and `libmkl_def.so` are in `/opt/arrayfire/lib/` .

Then I should use:  
`export LD_PRELOAD=/opt/arrayfire/lib/libmkl_avx2.so:/opt/arrayfire/lib/libmkl_def.so`

**However** , those files need `libmkl_sequential.so` and `libmkl_core.so` .

- `libmkl_core.so` is included on ArrayFire folder, so just use the same path as the others
- `libmkl_sequential.so` is not included with ArrayFire, but it does with Anaconda3 - in my computer is on the folder `/opt/anaconda3/lib/`

The final solution was :  
**`> export LD_PRELOAD=/opt/arrayfire/lib/libmkl_avx2.so:/opt/arrayfire/lib/libmkl_def.so:/opt/anaconda3/lib/libmkl_sequential.so:/opt/arrayfire/lib/libmkl_core.so`**  
`> julia`

```julia
julia> using ArrayFire
julia> ad = AFArray(rand(Float32,4,4))
julia> det(ad) # working !!

```

Then, after running Julia , operations like `inv` , `det` and `qr` are working