PyJulia, passing numpy array from Python side

New to Julia
, , ,
1

I’m looking into extending Python with Julia code through PyJulia, particularly allocating numpy arrays in Python, passing them to Julia (without copying) to do some processing and then use the result in Python again. This seems to be the reverse of what many other users are doing (i.e. using PyCall from Julia to call into Python), so maybe what I want is not supported.

But looking at the docs for PyCall, specifically GitHub - JuliaPy/PyCall.jl: Package to call Python functions from the Julia language, seems to suggest it is possible to pass a NumPy array from Python to Julia without copying:

Alternatively, the PyCall module also provides a new type PyArray (a subclass of AbstractArray ) which implements a no-copy wrapper around a NumPy array (currently of numeric types or objects only). Just use PyArray as the return type of a pycall returning an ndarray , or call PyArray(o::PyObject) on an ndarray object o . (Technically, a PyArray works for any Python object that uses the NumPy array interface to provide a data pointer and shape information.)

However, the referenced PyArray type lives on the Julia side so I can’t call it on the Python side. Using a Julia function with signature fn(x::PyArray) also doesn’t seem to do automatic conversion, as I get a no method matching fn(::Array{Float32,2}) error when called from Python with:

melis@juggle 15:33:~$ cat fn.jl 
function fn(x::PyArray)
    println("array size: $(size(x))");
    println("max element: $(maximum(x))")
    println("min element: $(minimum(x))")
    x[1,1] = 123
    return 2x
end

melis@juggle 15:33:~$ cat callit.py 
from julia.api import Julia
from julia import Main

jl = Julia(compiled_modules=False)
jl.eval('include("fn.jl")')

import numpy as np

x = np.array([[1,2,3], [4,5,6]], dtype=np.float64)

res = Main.fn(x)

melis@juggle 15:34:~$ python callit.py 
Traceback (most recent call last):
  File "callit.py", line 11, in <module>
    res = Main.fn(x)
RuntimeError: Julia exception: MethodError: no method matching fn(::Array{Float64,2})
Closest candidates are:
  fn(!Matched::PyArray) at /home/melis/concepts/blender-julia/fn.jl:1
Stacktrace:
 [1] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [2] invokelatest(::Any, ::Any) at ./essentials.jl:709
 [3] _pyjlwrap_call(::Function, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/melis/.julia/packages/PyCall/zqDXB/src/callback.jl:28
 [4] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/melis/.julia/packages/PyCall/zqDXB/src/callback.jl:49

Any other ways to make this work?

Edit: word

2

Why not remove the type specification on x in the fn definition? Just define fn(x) so that it works for any array-like object.

3

I don’t think that guarantees the array will not be copied

4

You could test if there is a copy e.g. by defining a numpy array with 1 billion entries (8 GB for float64) and passing it to Julia, while looking at your system memory consumption.
The most simple calling of Julia functions on Numpy arrays seems to make indeed a copy of them.

Test code:

import julia
jl = julia.Julia()
import numpy as np
my_sum = jl.eval("my_sum(x) = sum(x)")
small_a = np.random.randn(100)
my_sum(small_a) # compile method
a = np.random.randn(1_000_000_000) # 8 GB
np.sum(a)
my_sum(a) # memory consumtion increases significantly, indicating a copy
5

When I alter the array on the Julia side I also don’t see the change on the Python side when the call is done.

1 Like
6

I’m having exactly this problem, did you find a solution?

7

Just checked my test code from months ago. Using unsafe_wrap I apparently managed to get something working:

# alter_array.jl 
function fn(a)
    a[:] .= 9
end

function fn(addr, length)
    a = unsafe_wrap(Array{UInt32}, Ptr{UInt32}(addr), length)
    fn(a)
end

# t_alter_array.py 
import numpy
import julia
from julia.api import Julia

jl = Julia()
from julia import Main

jl.eval('include("alter_array.jl")')

a = numpy.array([1, 2, 3, 4, 5], 'uint32')
print(a)

Main.fn(a)
print(a)

addr = a.ctypes.data
length = a.shape[0]

Main.fn(addr, length)
print(a)

$ python t_alter_array.py 
[1 2 3 4 5]
[1 2 3 4 5]
[9 9 9 9 9]

Note that the call to Main.fn(a) does not result in the array being altered, it only works by passing the array address and length, i.e. Main.fn(addr, length).

1 Like
8

Thanks for the quick response. This worked.

9

Don’t do this. Just use pyfunction to declare the argument-type conversions that you want. That’s what it’s for.

Something like pyfn = Main.PyCall.pyfunction(Main.fn, Main.PyCall.PyArray) should tell it to pass the argument of pyfn as a PyArray, which is a no-copy wrapper.

10

Trying that now I get an error I remember seeing before:

$ cat t_alter_array.py
import numpy
import julia
from julia.api import Julia

jl = Julia()
from julia import Main

jl.eval('include("alter_array.jl")')

a = numpy.array([1, 2, 3, 4, 5], 'uint32')
print(a)

pyfn = Main.PyCall.pyfunction(Main.fn, Main.PyCall.PyArray)

pyfn(a)
print(a)

$ py t_alter_array.py
[1 2 3 4 5]
Traceback (most recent call last):
  File "/home/melis/concepts/blender-julia-test/test/t_alter_array.py", line 21, in <module>
    pyfn = Main.PyCall.pyfunction(Main.fn, Main.PyCall.PyArray)
  File "/home/melis/.local/lib/python3.9/site-packages/julia/core.py", line 176, in __getattr__
    return self.__try_getattr(name)
  File "/home/melis/.local/lib/python3.9/site-packages/julia/core.py", line 191, in __try_getattr
    if self._julia.isdefined(realname):
  File "/home/melis/.local/lib/python3.9/site-packages/julia/core.py", line 645, in isdefined
    raise ValueError(
ValueError: `julia.isdefined(name)` requires at least one dot in `name`.

I initially reported a similar error in the PyCall.jl repo, but you replied back then that is was a pyjulia error: Exception when trying to navigate Main.PyCall module · Issue #813 · JuliaPy/PyCall.jl · GitHub. Which I then subsequently reported in the pyjulia repo where it has been dormant marked “bug” ever since: Exception when trying to acces submodules of Main · Issue #414 · JuliaPy/pyjulia · GitHub.

So unless I’m missing something setting up the pyfunction as you suggest currently doesn’t seem possible. And that was perhaps the reason I tried the dirty hack passing the array address :wink:

11

I have no idea of what I’m doing, but pyfn = Main.eval("pyfunction(fn, PyArray)") seems to work for me.
On top of that it preserves the numpy data-types and dimensions correctly.

1 Like
12

That’s an even better workaround, nice

13

Plug: You might also like to try my package PythonCall.jl and it’s companion juliacall (which are similar to PyCall.jl and pyjulia) because there all mutable values are passed without copying by default.

1 Like
14

Could you show a specific example from Python to Julia, such as calling functions written in Python and returning to Julia? There is no similar example in the document. It’s not friendly for beginners of Julia.