Julia and NumPy are linked with MKL, which may cause conflicts and crashes

General Usage
1

I installed Julia 1.9.3 and made “]update” on Windows. This gave the following warning:

┌ Warning: both Julia and NumPy are linked with MKL, which may cause conflicts and crashes (#433).
└ @ PyCall D:\otter\home\.julia\packages\PyCall\ilqDX\src\numpy.jl:77

I never got such a warning previously.

I made some tests with PyPlot, and this worked.
Still I am concerned.

2

Not sure what’s going on here, but you should definitely consider using PythonCall.jl instead of PyCall.jl for increased reliability

3

This is a warning about this issue: Segfault with Julia MKL build (library conflict) · Issue #443 · JuliaPy/PyCall.jl · GitHub

4

I don’t think that’s relevant here. Shared library conflicts between Julia’s MKL (which uses the 64-bit API) and Numpy’s MKL (which uses the 32-bit API) will potentially happen no matter how you load Python. See also CBLAS/LAPACKE extension for 64bit integer · Issue #666 · Reference-LAPACK/lapack · GitHub

However, maybe the issue has gone away? Various Julia people bugged Intel for a long while about versioning their symbols, and I vaguely seem to recall that they finally fixed it so that you could link both 32- and 64-bit versions simultaneously?

(If so you can ignore the warning, which should probably be removed for recent versions.)

4 Likes
5

FWIW I had the opposite experience and also real battle tested pkgs from python side are both using PyCall still (pysr, diffeq)

6

diffeqpy is now using PythonCall/JuliaCall and it fixed a lot of the issues. It even has a specialized JIT and supports GPUs now.

1 Like
7

Thanks for the information about PythonCall - I did not know this.
In this particular case, this does not help, because PyPlot has PyCall in its Project.toml file.
Yesterday, I tested more and an even more critical issue appeared:
When running the unit tests of Modia3D with PyPlot plotting enabled, Julia terminates during the tests with the following error message due to OpenMP (if PyPlot is not called, this crash does not occur):

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized. 
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the 
program. That is dangerous, since it can degrade performance or cause incorrect results. The 
est thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by 
avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported,
undocumented workaround you can set the environment variable
KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may
cause crashes or silently produce incorrect results. For more information, please see
http://www.intel.com/software/products/support/.

I guess, OpenMP is called during simulation of a larger Modia3D model and in PyPlot OpenMP is called as well. Any idea what to do?

It is strange that this did not happen before, so either in Julia or in PyPlot something has changed in the last months.

Note, we are using PyPlot because it is much better for line plots (that we need a lot) with respect to Makie: PyPlot supports multiple windows, the windows have a toolbar with useful functionality (e.g. exporting plot in various formats such as pdf), the quality of the text fonts is better and the reaction is always fast (no startup time etc.).

8

(PythonPlot is an equivalent of PyPlot that uses PythonCall.)

PyPlot itself does not call OpenMP. Apparently you are calling two different libraries that link two different versions of OpenMP, but I’m not sure what those libraries might be. (They could be two Python packages for all I know.) What else are you using besides PyPlot?

9

Thanks for the hint to PythonPlot. I did not know this. We will try it.

PyPlot itself does not call OpenMP. Apparently you are calling two different libraries that link two different versions of OpenMP, but I’m not sure what those libraries might be. (They could be two Python packages for all I know.) What else are you using besides PyPlot?

Last week: runtests.jl of Modia3D was working as expected, as long as PyPlot calls were disabled. If PyPlot was called in runtests.jl, for some simulations the problem with OpenMP occured.

Some days ago, I made a new version of Modia with support for GLMakie 0.8 (previously, it was for version 0.6). If I then switched to use PyPlot, everything worked with Modia3D and the error with OpenMP was gone. I don’t understand this, because the GLMakie part is completely unrelated.

Anyway, I am closing this topic, because the problem does no longe occur.