Intel Mkl Windows building

Hi, catching up on stuff now.

The current state of things is that you can ] add https://github.com/arnavs/ArpackMKL.jl on Linux machines no problem.

OS support is provided by https://github.com/arnavs/ArpackMKLBuilder, which provides the MKL-built Arpack binaries for use in the main (“consumer-facing”) package.

On Windows, I was having trouble with the .dll setup vs the .dylib, .so which is used in Linux. This whole thing is also above my pay grade technically, which makes it hard to debug.

4 Likes

A couple of days ago, I requested a pull to their readme tutorial adding those commands we found out from this thread. However, they have not accepted that yet.
https://github.com/JuliaComputing/MKL.jl/pull/8

1 Like

I gave this a go on Windows, and here’s the step of the build script which failed. I’m looking at the ArpackBuilder script though, and there isn’t any Windows-specific fudging. So this should be possible in principle.

The specific error looks like it’s using -rdynamic, which should be replaced with --export-dynamic in later gcc versions? Not sure. Perhaps @andreasnoack can advise.

sandbox:$WORKSPACE/srcdir/arpack-ng/build# cmake ..  -DINTERFACE64=1 -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-L$MKLROOT/lib -lmkl_rt" -DLAPACK_LIBRARIES="-L$MKLROOT/lib -lmkl_rt" -DCMAKE_EXE_LINKER_FLAGS="${CMAKE_EXE_LINKER_FLAGS}" -DCMAKE_Fortran_FLAGS="${FFLAGS}"
-- The C compiler identification is GNU 4.8.5
-- The Fortran compiler identification is GNU 4.8.5
-- Check for working C compiler: /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc
-- Check for working C compiler: /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc -- broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/bin/make" "cmTC_d442d/fast"
    /usr/bin/make -f CMakeFiles/cmTC_d442d.dir/build.make CMakeFiles/cmTC_d442d.dir/build
    make[1]: Entering directory '/workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o
    /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc    -o CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o   -c /workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_d442d
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_d442d.dir/link.txt --verbose=1
    /opt/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-gcc      -rdynamic CMakeFiles/cmTC_d442d.dir/testCCompiler.c.o  -o cmTC_d442d 
    x86_64-w64-mingw32-gcc: error: unrecognized command line option '-rdynamic'
    make[1]: *** [CMakeFiles/cmTC_d442d.dir/build.make:87: cmTC_d442d] Error 1
    make[1]: Leaving directory '/workspace/srcdir/arpack-ng/build/CMakeFiles/CMakeTmp'
    make: *** [Makefile:126: cmTC_d442d/fast] Error 2
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:7 (project)
1 Like

@dawbarton. Thanks I have been able to build Julia with MKL. I am using Julia 1.1.1 on windows 10.
My problem is that Julia startup take almost 10s. Without MKL, it is just fraction of a second or two.

Is there anything I need to do? I have noticed that first julia startup after MKL build takes around 4s but subsequent startup become slower. Can not figure out the reason.

The problem is that PackageCompiler rebuilds the Julia system image without including the relevant REPL statements. If I understand the code correctly, the file https://github.com/JuliaLang/julia/blob/master/contrib/generate_precompile.jl generates all the necessary precompile statements for the REPL and it’s just a case of including it as part of the PackageCompiler rebuild. I did try doing this on Windows but I wasn’t successful (some problems with Windows not having pseudo-TTYs if I remember correctly) and I didn’t dig any further since using the MKL isn’t high on my priority list.

1 Like

Why not just build Julia and link it against MKL (instead of using MKL.jl)? Building Julia is really easy. Even on Windows it’s not that difficult. See here for instructions.

1 Like

Thanks

That’s not true! It is not easy to build. See the spoiler in the OP for my original question regarding problems in building the Julia with MKL.

For now, MKL.jl has solved my issue. Julia runs fast enough. It is not 10 Seconds! Maybe 1-2 seconds.
The problem is that the repository is inactive. Neither they merge pull requests (see my PR) nor they respond to the issues (see my issue). That happens when 8 people manage 50 repository. Lack of people!

I’m sorry but your OP confuses me - it is a mixture of MKL.jl and building Julia from scratch with non-permalinks.

What issues are you facing with the instructions I linked? To be concrete, these windows instructions.

Personally, I always found MKL.jl pretty much useless because of its limitations - and somewhat hacky.

This is my old question, which led to using MKL.jl instead.
Here it says to set your environment. How should I do that on windows?
https://github.com/JuliaLang/julia/blob/master/doc/build/build.md#intel-mkl
Also, by Make.user does it mean ‘Make.inc’?

I followed these until before building:
https://github.com/JuliaLang/julia/blob/master/doc/build/windows.md


Also, I have not tested MKL.jl fully, but it shows some improvement in some cases for speed as I showed here in my repository: Julia-Matlab-Benchmark


But not for all the operations. Maybe this what you are calling its limitations.

I don’t think that reposting parts of you OP helps to reduce the confusion. Let me try to address the points nonetheless.

You should read the Windows instructions if you try to build Julia on Windows. But no, Make.user means Make.user not Make.inc.

There is no doubt that Julia + MKL has, in many cases, superior performance over Julia + OpenBLAS (see for example my matrix multiplication benchmark here). By limitations I’m refering to what is listed as limitations on MKL.jl’s github page, in particular the first point.

Again, if you have issues building Julia on Windows, it’d be great if you could share what those issues are. This way I (or someone else) might be able to provide help.

2 Likes

Well I have seen that link! It is not straight forward. Instructions for building with MKL is dispersed around the documentation.

Just compare this Python which ships with MKL powered libraries!!
https://docs.anaconda.com/mkl-optimizations/

I prefer binaries or at least automated building regarding its limitations, and I know a lot of users are the same as me!

1 Like

The legal issues have been explained dozens of times and have not changed. I won’t waste breath by repeating them yet again.

4 Likes

Found nice trick to maximize MKL performance on AMD CPU’s based on Agner Fog findings: Anaconda on Windows with MKL Support.

Pay attnetion to the compilation trick: https://github.com/fo40225/Anaconda-Windows-AMD/blob/master/site.cfg. Just incorporating Agner Fog’s CPU Dispatch.

1 Like