Intel Mkl Windows building

Try

]add https://github.com/JuliaComputing/MKL.jl

Hopefully that works - don’t have my laptop to check.

1 Like

This took some time to complete, and seemed to do something.

(v1.1) pkg> add https://github.com/JuliaComputing/MKL.jl
  Updating registry at `C:\Users\username\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `https://github.com/JuliaComputing/MKL.jl`
  Updating git-repo `https://github.com/JuliaComputing/MKL.jl` Resolving package versions...
 Installed HTTPClient ────── v0.2.1
 Installed Libz ──────────── v1.0.0
 Installed WinRPM ────────── v0.4.2
 Installed LibCURL ───────── v0.5.1
 Installed TextWrap ──────── v0.3.0
 Installed BufferedStreams ─ v1.0.0
 Installed PackageCompiler ─ v0.6.3
 Installed WebIO ─────────── v0.8.4
 Installed LibExpat ──────── v0.5.0
 Installed BinDeps ───────── v0.8.10
 Installed ArgParse ──────── v0.6.2
  Updating `C:\Users\username\.julia\environments\v1.1\Project.toml`
  [33e6dc65] + MKL v0.0.0 #master (https://github.com/JuliaComputing/MKL.jl)
  Updating `C:\Users\username\.julia\environments\v1.1\Manifest.toml`
  [c7e460c6] + ArgParse v0.6.2
  [9e28174c] + BinDeps v0.8.10
  [e1450e63] + BufferedStreams v1.0.0
  [0862f596] + HTTPClient v0.2.1
  [b27032c2] + LibCURL v0.5.1
  [522f3ed2] + LibExpat v0.5.0
  [2ec943e9] + Libz v1.0.0
  [33e6dc65] + MKL v0.0.0 #master (https://github.com/JuliaComputing/MKL.jl)
  [9b87118b] + PackageCompiler v0.6.3
  [b718987f] + TextWrap v0.3.0
  [0f1e0344] ↑ WebIO v0.8.1 ⇒ v0.8.4
  [c17dfb99] + WinRPM v0.4.2
  Building WebIO ──────────→ `C:\Users\username\.julia\packages\WebIO\kk2q1\deps\build.log`
  Building LibCURL ────────→ `C:\Users\username\.julia\packages\LibCURL\teHPG\deps\build.log`
  Building WinRPM ─────────→ `C:\Users\username\.julia\packages\WinRPM\Y9QdZ\deps\build.log`
  Building PackageCompiler → `C:\Users\username\.julia\packages\PackageCompiler\oT98U\deps\build.log`
  Building MKL ────────────→ `C:\Users\username\.julia\packages\MKL\36eeJ\deps\build.log`

However, when I ran this code after it still says it is openBlas.

using LinearAlgebra
 BLAS.vendor()

and now when I restart my Julia this is the result:
Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.

Could you explain what do you mean by: The loss of arpack and suitesparse functionality will take more time, but there is progress.?

1 Like

As far as I understand it, if you use mkl.jl and package compile it into your system image, it works great for almost all of base and is a solid experience.

The current issue is that other packages which assume that openblas is installed need to be patched to support using mkl for any dense calculations.

In particular, Sparse Arrays · The Julia Language uses suitesparse as matrix factorization lfor linear solves (I think) and arpack.jl currently requires openblas (necessary for eigs).People are trying to work through the binary builder issues to make usage with mkl seamless,but I would guess later in July

1 Like

Yes, this what should happen; the new sysimg will only take effect once Julia is restarted.

That means the required dlls are not on the Windows path. You need to add the location that it was installed to your path - for me that is C:\Users\dawbarton\.julia\packages\MKL\36eeJ\deps\usr\bin. After that it should all work (at least it does for me!).

3 Likes

Thank you! It finally worked. I added that folder to my user Path in environment variables, and it worked!

julia> using LinearAlgebra
julia>  BLAS.vendor()
:mkl

Hi, catching up on stuff now.

The current state of things is that you can ] add https://github.com/arnavs/ArpackMKL.jl on Linux machines no problem.

OS support is provided by https://github.com/arnavs/ArpackMKLBuilder, which provides the MKL-built Arpack binaries for use in the main (“consumer-facing”) package.

On Windows, I was having trouble with the .dll setup vs the .dylib, .so which is used in Linux. This whole thing is also above my pay grade technically, which makes it hard to debug.

4 Likes

A couple of days ago, I requested a pull to their readme tutorial adding those commands we found out from this thread. However, they have not accepted that yet.
https://github.com/JuliaComputing/MKL.jl/pull/8

1 Like

I gave this a go on Windows, and here’s the step of the build script which failed. I’m looking at the ArpackBuilder script though, and there isn’t any Windows-specific fudging. So this should be possible in principle.

The specific error looks like it’s using -rdynamic, which should be replaced with --export-dynamic in later gcc versions? Not sure. Perhaps @andreasnoack can advise.

sandbox:$WORKSPACE/srcdir/arpack-ng/build# cmake ..  -DINTERFACE64=1 -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-L$MKLROOT/lib -lmkl_rt" -DLAPACK_LIBRARIES="-L$MKLROOT/lib -lmkl_rt" -DCMAKE_EXE_LINKER_FLAGS="${CMAKE_EXE_LINKER_FLAGS}" -DCMAKE_Fortran_FLAGS="${FFLAGS}"
-- The C compiler identification is GNU 4.8.5
-- The Fortran compiler identification is GNU 4.8.5
-- Check for working C compiler: /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc
-- Check for working C compiler: /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc -- broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/bin/make" "cmTC_d442d/fast"
    /usr/bin/make -f CMakeFiles/cmTC_d442d.dir/build.make CMakeFiles/cmTC_d442d.dir/build
    make[1]: Entering directory '/workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o
    /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc    -o CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o   -c /workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_d442d
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_d442d.dir/link.txt --verbose=1
    /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc      -rdynamic CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o  -o cmTC_d442d 
    x86_64-w64-mingw32-gcc: error: unrecognized command line option '-rdynamic'
    make[1]: *** [CMakeFiles/cmTC_d442d.dir/build.make:87: cmTC_d442d] Error 1
    make[1]: Leaving directory '/workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp'
    make: *** [Makefile:126: cmTC_d442d/fast] Error 2
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:7 (project)
1 Like

@dawbarton. Thanks I have been able to build Julia with MKL. I am using Julia 1.1.1 on windows 10.
My problem is that Julia startup take almost 10s. Without MKL, it is just fraction of a second or two.

Is there anything I need to do? I have noticed that first julia startup after MKL build takes around 4s but subsequent startup become slower. Can not figure out the reason.

The problem is that PackageCompiler rebuilds the Julia system image without including the relevant REPL statements. If I understand the code correctly, the file https://github.com/JuliaLang/julia/blob/master/contrib/generate_precompile.jl generates all the necessary precompile statements for the REPL and it’s just a case of including it as part of the PackageCompiler rebuild. I did try doing this on Windows but I wasn’t successful (some problems with Windows not having pseudo-TTYs if I remember correctly) and I didn’t dig any further since using the MKL isn’t high on my priority list.

1 Like

Why not just build Julia and link it against MKL (instead of using MKL.jl)? Building Julia is really easy. Even on Windows it’s not that difficult. See here for instructions.

1 Like

Thanks

That’s not true! It is not easy to build. See the spoiler in the OP for my original question regarding problems in building the Julia with MKL.

For now, MKL.jl has solved my issue. Julia runs fast enough. It is not 10 Seconds! Maybe 1-2 seconds.
The problem is that the repository is inactive. Neither they merge pull requests (see my PR) nor they respond to the issues (see my issue). That happens when 8 people manage 50 repository. Lack of people!

I’m sorry but your OP confuses me - it is a mixture of MKL.jl and building Julia from scratch with non-permalinks.

What issues are you facing with the instructions I linked? To be concrete, these windows instructions.

Personally, I always found MKL.jl pretty much useless because of its limitations - and somewhat hacky.

This is my old question, which led to using MKL.jl instead.
Here it says to set your environment. How should I do that on windows?
https://github.com/JuliaLang/julia/blob/master/doc/build/build.md#intel-mkl
Also, by Make.user does it mean ‘Make.inc’?

I followed these until before building:
https://github.com/JuliaLang/julia/blob/master/doc/build/windows.md


Also, I have not tested MKL.jl fully, but it shows some improvement in some cases for speed as I showed here in my repository: Julia-Matlab-Benchmark


But not for all the operations. Maybe this what you are calling its limitations.

I don’t think that reposting parts of you OP helps to reduce the confusion. Let me try to address the points nonetheless.

You should read the Windows instructions if you try to build Julia on Windows. But no, Make.user means Make.user not Make.inc.

There is no doubt that Julia + MKL has, in many cases, superior performance over Julia + OpenBLAS (see for example my matrix multiplication benchmark here). By limitations I’m refering to what is listed as limitations on MKL.jl’s github page, in particular the first point.

Again, if you have issues building Julia on Windows, it’d be great if you could share what those issues are. This way I (or someone else) might be able to provide help.

2 Likes

Well I have seen that link! It is not straight forward. Instructions for building with MKL is dispersed around the documentation.

Just compare this Python which ships with MKL powered libraries!!
https://docs.anaconda.com/mkl-optimizations/

I prefer binaries or at least automated building regarding its limitations, and I know a lot of users are the same as me!

1 Like

The legal issues have been explained dozens of times and have not changed. I won’t waste breath by repeating them yet again.

4 Likes

Found nice trick to maximize MKL performance on AMD CPU’s based on Agner Fog findings: Anaconda on Windows with MKL Support.

Pay attnetion to the compilation trick: https://github.com/fo40225/Anaconda-Windows-AMD/blob/master/site.cfg. Just incorporating Agner Fog’s CPU Dispatch.

1 Like