# Questions about pyjulia usage

**URL:** <https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332>\
**Category:** General Usage\
**Tags:** pyjulia\
**Created:** [November 11, 2021, 2:31pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332 "2021-11-11T14:31:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 11, 2021, 2:31pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/1 "2021-11-11T14:31:12Z")

</div>

I am trying to see how well a code runs called from python, and I’m facing some problems and have some doubts:

1. When trying to import any package I am getting this error:

```julia
Your Python interpreter "/usr/bin/python3"
is statically linked to libpython. Currently, PyJulia does not fully
support such Python interpreter.

```

> **details**
>
> ```julia
> leandro@avell:~% ipython3 
> Python 3.8.10 (default, Sep 28 2021, 16:10:42) 
> Type 'copyright', 'credits' or 'license' for more information
> IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
> 
> In [1]: from julia import CellListMap                                                    
> ---------------------------------------------------------------------------
> UnsupportedPythonError Traceback (most recent call last)
> <ipython-input-1-7997273b03be> in <module>
> ----> 1 from julia import CellListMap
> 
> /usr/lib/python3.8/importlib/_bootstrap.py in _find_and_load(name, import_)
> 
> /usr/lib/python3.8/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)
> 
> /usr/lib/python3.8/importlib/_bootstrap.py in _load_unlocked(spec)
> 
> /usr/lib/python3.8/importlib/_bootstrap.py in _load_backward_compatible(spec)
> 
> ~/.local/lib/python3.8/site-packages/julia/core.py in load_module(self, fullname)
> 246 return sys.modules.setdefault(fullname,
> 247 JuliaMainModule(self, fullname))
> --> 248 elif self.julia.isafunction(juliapath):
> 249 return self.julia.eval(juliapath)
> 250 
> 
> ~/.local/lib/python3.8/site-packages/julia/core.py in julia(self)
> 237 @property
> 238 def julia(self):
> --> 239 self. __class__.julia = julia = Julia()
> 240 return julia
> 241 
> 
> ~/.local/lib/python3.8/site-packages/julia/core.py in __init__ (self, init_julia, jl_init_path, runtime, jl_runtime_path, debug, **julia_options)
> 481 logger.debug("compiled_modules = %r", options.compiled_modules)
> 482 if not (options.compiled_modules == "no" or is_compatible_python):
> --> 483 raise UnsupportedPythonError(jlinfo)
> 484 
> 485 self.api.init_julia(options)
> 
> UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.
> 
> Julia executable:
> julia
> Python interpreter and libpython used by PyCall.jl:
> /usr/bin/python3
> /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
> Python interpreter used to import PyJulia and its libpython.
> /usr/bin/python3
> /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
> 
> Your Python interpreter "/usr/bin/python3"
> is statically linked to libpython. Currently, PyJulia does not fully
> support such Python interpreter.
> 
> The easiest workaround is to pass `compiled_modules=False` to `Julia`
> constructor. To do so, first *reboot* your Python REPL (if this happened
> inside an interactive session) and then evaluate:
> 
> >>> from julia.api import Julia
> >>> jl = Julia(compiled_modules=False)
> 
> Another workaround is to run your Python script with `python-jl`
> command bundled in PyJulia. You can simply do:
> 
> $ python-jl PATH/TO/YOUR/SCRIPT.py
> 
> See `python-jl --help` for more information.
> 
> For more information, see:
> 
> https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
> 
> ```

The workaround proposed there (i. e. `jl = Julia(compiled_modules=False)`) makes the code work. However, since the performance I’m getting is not very good, this might not be a solution.

1. My package works on `(3,N)` matrices. In Julia I create them with `rand(3,1000)`, for example. Since Python is row-major, I thought that with numpy that would correspond to a `np.random.random((1000,3))` array. However the julia package works with the `np.random.random((3,1000))` array, which confuses me. Is a copy being made that transposes the array in the interface of `pyjulia`? edit: this is explicitly mentioned [here](https://discourse.julialang.org/t/a-stack-of-inverse-matrices-calculation-when-comparing-to-python/71318/3), but, as shown bellow, does not seem to be reason for the difference in performance.

2. How to allow Julia to run in parallel from within python? ~~I didn’t find how to do it.~~ Found the solution at the unnatural [no threading support](https://pyjulia.readthedocs.io/en/latest/limitations.html#no-threading-support) section of the manual (yet, although the `nthreads()` command indicates that more threads are available to Julia, the julia package itself appears not to be using them).

3. Finally, even when running in serial, the execution of a calculation is taking almost 10 times longer when the package is called from python than from within Julia. The time required even to copy the input arrays and transpose them (if that is the case) does not seem to justify the difference. Any hint of what else may be going?

A MWE example in Julia is:

```julia
julia> using CellListMap

julia> x = rand(3,50000);

julia> @btime CellListMap.neighbourlist(x,0.05,parallel=false);
  59.951 ms (17598 allocations: 32.85 MiB)

```

and what I am able to run in `IPython` is:

```julia
In [4]: import numpy as np                                                      

In [5]: from julia import Julia                                                 

In [6]: jl = Julia(compiled_modules=False)                                      

In [7]: from julia import CellListMap                                           

In [8]: x = np.random.random((3,50000)); 

In [9]: %timeit CellListMap.neighbourlist(x,0.05);                             
619 ms ± 84 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [10]: %timeit CellListMap.neighbourlist(x,0.05);                             
608 ms ± 113 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 11, 2021, 2:43pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/2 "2021-11-11T14:43:04Z")

</div>

The alternative to use the `python-jl -m IPython` solves the first problem (though I am not sure how python users will find this acceptable, any experience with that?).

The performance is not good, anyway (only slightly better, but far from the performance of the call directly from Julia).

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 11, 2021, 5:48pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/3 "2021-11-11T17:48:50Z")

</div>

The memory layout does not seem to be the cause of the slowdown, because I can create both layouts using a julia or a numpy call, and the peformance is the same (and still ~8-10 slower than running the same function in Julia):

```julia
In [76]: x = %julia rand(3,50000)                                                                                        

In [77]: x.flags                                                                                                         
Out[77]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

In [78]: %timeit CellListMap.neighbourlist(x,0.05)                                                                       
416 ms ± 66.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [79]: x = np.random.random((3,50000))                                                                                 

In [80]: x.flags                                                                                                         
Out[80]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

In [81]: %timeit CellListMap.neighbourlist(x,0.05)                                                                       
419 ms ± 66.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [82]:

```

In Julia:

```julia
julia> x = rand(3,50000);

julia> @btime CellListMap.neighbourlist($x,0.05,parallel=false);
  52.811 ms (17640 allocations: 32.86 MiB)

```

The first-time call of the function takes ~4s in Julia, thus that slowdown cannot be caused by repeated compilation of the code by each julia call (or the `%timeit` macro makes it compile only once, but every time).

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [November 11, 2021, 6:28pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/4 "2021-11-11T18:28:39Z")

</div>

I’ve had very bad experiences with pyjulia. Most prominently unexplained and irregular crashes. (Segfaults? I don’t remember) - my memory is that there just isn’t enough interest in calling Julia from Python. We are therefore rewriting all interfaces via embedding Julia in C and then linking to other libraries including ASE and OpenMM

Since we are interested in the same application domain I think there would be a lot of scope to collaborate if you are interested.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 11, 2021, 6:37pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/5 "2021-11-11T18:37:29Z")

</div>

I had already in these tests some out of memory errors. Not fun.

I was playing with the possibility of calling the package from python, because for some applications it is working faster than the `scipy` alternative (thus it could be interesting to use it). But I don’t have any application for that myself, for the moment.

For example, if I run that test with a matrix of `(3,500_000)` twice the python section gets killed. In Julia I can run problems with as much as 10 million columns with no problem.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [November 11, 2021, 10:11pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/6 "2021-11-11T22:11:08Z")

</div>

That’s odd, any conversion overheads should be negligible compared to calling that function.

You may like to try [JuliaCall](https://pypi.org/project/juliacall/) instead.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 11, 2021, 10:33pm UTC](https://discourse.julialang.org/t/questions-about-pyjulia-usage/71332/7 "2021-11-11T22:33:15Z")

</div>

I found my way around it (great package, thanks… the documentation needs some work 🙂 ).

Now I have:

```python
In [2]: from juliacall import Main as jl

In [3]: jl.Pkg.add("CellListMap")                                                                    
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
  No Changes to `~/.julia/environments/PythonCall/Project.toml`
  No Changes to `~/.julia/environments/PythonCall/Manifest.toml`

In [5]: jl.seval("using CellListMap")                                                                

In [7]: import numpy as np                                                                           

In [9]: x = np.random.random((3,50000))                                                              

In [10]: %timeit jl.CellListMap.neighbourlist(x,0.05)                                                
60.5 ms ± 5.59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

which has no perceptible overhead.

Compared to `scipy.KDtree`:

```python
In [8]: from scipy.spatial import KDTree

In [11]: y = np.transpose(x)                                                                         

In [12]: def pp(points,r) :  
    ...: kd_tree = KDTree(points)  
    ...: pairs = kd_tree.query_pairs(r)  
    ...: return pairs 
    ...:                                                                                             

In [13]: %timeit pp(y,0.05)                                                                          
213 ms ± 1.17 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [20]: len(jl.CellListMap.neighbourlist(x,0.05))                                                   
Out[20]: 618076

In [21]: len(pp(y,0.05))                                                                             
Out[21]: 618076

```

I have only to find out how to run julia multi-threaded.