I am trying to see how well a code runs called from python, and I’m facing some problems and have some doubts:
- When trying to import any package I am getting this error:
Your Python interpreter "/usr/bin/python3" is statically linked to libpython. Currently, PyJulia does not fully support such Python interpreter.
leandro@avell:~% ipython3 Python 3.8.10 (default, Sep 28 2021, 16:10:42) Type 'copyright', 'credits' or 'license' for more information IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help. In : from julia import CellListMap --------------------------------------------------------------------------- UnsupportedPythonError Traceback (most recent call last) <ipython-input-1-7997273b03be> in <module> ----> 1 from julia import CellListMap /usr/lib/python3.8/importlib/_bootstrap.py in _find_and_load(name, import_) /usr/lib/python3.8/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_) /usr/lib/python3.8/importlib/_bootstrap.py in _load_unlocked(spec) /usr/lib/python3.8/importlib/_bootstrap.py in _load_backward_compatible(spec) ~/.local/lib/python3.8/site-packages/julia/core.py in load_module(self, fullname) 246 return sys.modules.setdefault(fullname, 247 JuliaMainModule(self, fullname)) --> 248 elif self.julia.isafunction(juliapath): 249 return self.julia.eval(juliapath) 250 ~/.local/lib/python3.8/site-packages/julia/core.py in julia(self) 237 @property 238 def julia(self): --> 239 self.__class__.julia = julia = Julia() 240 return julia 241 ~/.local/lib/python3.8/site-packages/julia/core.py in __init__(self, init_julia, jl_init_path, runtime, jl_runtime_path, debug, **julia_options) 481 logger.debug("compiled_modules = %r", options.compiled_modules) 482 if not (options.compiled_modules == "no" or is_compatible_python): --> 483 raise UnsupportedPythonError(jlinfo) 484 485 self.api.init_julia(options) UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported. Julia executable: julia Python interpreter and libpython used by PyCall.jl: /usr/bin/python3 /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 Python interpreter used to import PyJulia and its libpython. /usr/bin/python3 /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 Your Python interpreter "/usr/bin/python3" is statically linked to libpython. Currently, PyJulia does not fully support such Python interpreter. The easiest workaround is to pass `compiled_modules=False` to `Julia` constructor. To do so, first *reboot* your Python REPL (if this happened inside an interactive session) and then evaluate: >>> from julia.api import Julia >>> jl = Julia(compiled_modules=False) Another workaround is to run your Python script with `python-jl` command bundled in PyJulia. You can simply do: $ python-jl PATH/TO/YOUR/SCRIPT.py See `python-jl --help` for more information. For more information, see: https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
The workaround proposed there (i. e.
jl = Julia(compiled_modules=False)) makes the code work. However, since the performance I’m getting is not very good, this might not be a solution.
My package works on
(3,N)matrices. In Julia I create them with
rand(3,1000), for example. Since Python is row-major, I thought that with numpy that would correspond to a
np.random.random((1000,3))array. However the julia package works with the
np.random.random((3,1000))array, which confuses me. Is a copy being made that transposes the array in the interface of
pyjulia? edit: this is explicitly mentioned here, but, as shown bellow, does not seem to be reason for the difference in performance.
How to allow Julia to run in parallel from within python?
I didn’t find how to do it.Found the solution at the unnatural no threading support section of the manual (yet, although the
nthreads()command indicates that more threads are available to Julia, the julia package itself appears not to be using them).
Finally, even when running in serial, the execution of a calculation is taking almost 10 times longer when the package is called from python than from within Julia. The time required even to copy the input arrays and transpose them (if that is the case) does not seem to justify the difference. Any hint of what else may be going?
A MWE example in Julia is:
julia> using CellListMap julia> x = rand(3,50000); julia> @btime CellListMap.neighbourlist(x,0.05,parallel=false); 59.951 ms (17598 allocations: 32.85 MiB)
and what I am able to run in
In : import numpy as np In : from julia import Julia In : jl = Julia(compiled_modules=False) In : from julia import CellListMap In : x = np.random.random((3,50000)); In : %timeit CellListMap.neighbourlist(x,0.05); 619 ms ± 84 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In : %timeit CellListMap.neighbourlist(x,0.05); 608 ms ± 113 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)