JuliaCall: Pass numpy array to julia function as Vector{Float64}

General Usage
, , , ,
1

I have recently started using JuliaCall to use Julia functionality in python, but I am having trouble understanding how type conversions work, and I have the following problem.

  • I have a julia function which has an argument whose type is specified as Vector{Float64}.
  • I have a list of floats in python, and I would like to call the julia function on that list
  • When I do that, I get a MethodError because the types don’t match.
  • The error also happens when I use a numpy array instead of a list

Here is a minimal working example.

File testing_juliacall.jl:

function myfunc(a::Vector{Float64})
    return
end

File testing_julicall.py:

from juliacall import Main as jl

pylist = [1.0, 2.0, 3.0]
jl.include("testing_juliacall.jl")
jl.vector_func(pylist)

File testing_juliacall_numpy.py:

from juliacall import Main as jl
import numpy as np

numpy_array = np.array([1.0, 2.0, 3.0])
jl.include("testing_juliacall.jl")
jl.vector_func(numpy_array)

Results:

$ python testing_juliacall.py 
Traceback (most recent call last):
  File "testing_juliacall.py", line 5, in <module>
    jl.vector_func(pylist)
  File "/home/spencer/.julia/packages/PythonCall/DqZCE/src/jlwrap/any.jl", line 201, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
TypeError: Julia: MethodError: no method matching vector_func(::PythonCall.PyList{PythonCall.Py})
Closest candidates are:
  vector_func(!Matched::Vector{Float64}) at ~/LLNL/JuqPy/testing_juliacall.jl:1

$ python testing_juliacall_numpy.py 
Traceback (most recent call last):
  File "testing_juliacall_numpy.py", line 6, in <module>
    jl.vector_func(numpy_array)
  File "/home/spencer/.julia/packages/PythonCall/DqZCE/src/jlwrap/any.jl", line 201, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
TypeError: Julia: MethodError: no method matching vector_func(::PythonCall.PyArray{Float64, 1, true, true, Float64})
Closest candidates are:
  vector_func(!Matched::Vector{Float64}) at ~/LLNL/JuqPy/testing_juliacall.jl:1

Any advice on how to get this working? Does JuliaCall just not play very well with specified types? Would I have to write another method of vector_func which accepts objects of type PythonCall.PyList{PythonCall.Py} or PythonCall.PyArray{Float64, 1, true, true, Float64}? That would not be as convenient as I hoped.

2

Perhaps I am missing the main point, but the function isn’t called vector_func, but myfunc.

1 Like
3

If you type the function for AbstractVector it should work.

But the Julia vector and the numpy vector are different, and some conversion is needed to have a better interoperability. I have been dealing with this recently, and ended writing a small auxiliary function (in the Julia side) to copy between the types (see CellListMap.jl/CellListMap.py at main · m3g/CellListMap.jl · GitHub).

Trying to copy on the Python side completely kills the performance.

4

This is generally a mistake. A big advantage of using a high-level language is the ability to write functions that work on more general types. Can you broaden your function to accept e.g. AbstractArray{<:Real}?

Failing that, you can in principle wrap a numpy 1d array with a Julia Vector of the same element type without copying the data by using the unsafe_wrap function.

1 Like
5

This is a nice example, so I’ll run through the options in detail.

To summarise the problem, you have defined a Julia function myfunc(::Vector{Float64}) and then find that you cannot call it from Python (using JuliaCall) with a list or numpy.ndarray as the argument. This fails because of the default conversion rules when calling a Julia function from Python:

  • list is converted to PyList{Py}
  • numpy.ndarray is converted to PyArray{Float64,1} (in this case)

Option 1 (highly recommended): As suggested in previous replies, make the signature of myfunc more general, e.g. myfunc(::AbstractArray{<:Real}). This is generally a good thing to do in Julia any, but in this case it means that it can be called with a PyArray{Floay64,1} argument, so will work with numpy.ndarray or any other Python array type. However it will still not work with list because PyList{Py} <: AbstractVector{<:Real} is false (because Py <: Real is false).

Option 2: If you can’t do that, then write a Julia wrapper function

myfunc2(a::AbstractVector) = myfunc(convert(Vector{Float64}, a))

and call that instead.

Option 3: If you have done 2 or 3, then you can make it also work with lists by explicitly converting to a numpy array from Python:

jl.myfunc(np.asarray([1.0, 2.0, 3.0]))

Option 4: Or without any changes to myfunc you could instead explicitly convert to a Vector{Float64} from Python:

jl.myfunc(juliacall.convert(jl.Vector[jl.Float64], [1.0, 2.0, 3.0]))

Option 5: Or you can create a wrapper function which does the conversion on the Julia side:

myfunc = jl.seval("pyfunc((a::Py)->myfunc(pyconvert(Vector{Float64}, a)))")

To explain this a bit, pyfunc wraps a Julia function into a Python function, but the arguments are not automatically converted (hence the a::Py argument) and so you can call pyconvert to convert them to the desired type. This option is mostly useful from the Julia side to create a Python callback function.

If I were you I’d do Option 1 and maybe Option 3. Or do Option 4.

4 Likes
6

IIUC you are unzipping a Julia vector of 3-tuples into 3 numpy arrays. You can do this from Python using numpy’s field indexing:

x = ... # Julia array of 3-tuples
x = np.asarray(x) # now it's a numpy.ndarray of 3-tuples
x0 = x["f0"] # numpy.ndarray of the 1st field
x1 = x["f1"] # 2nd field
x2 = x["f2"] # 3rd field

These are all non-copying views into the original x, so should be really fast.

7

Can’t check now, does that work if the types of the fields differ? (Which is my case there, Tuple{Int,Int,Float64})

8

For PyArray (a numpy array) you should also be able to use a copy-free wrapper, no? Something like:

myfunc2(a::PyArray{Float64,1}) =
    GC.@preserve a myfunc(unsafe_wrap(Array, pointer(a), length(a)))
9

Yes that works fine - such an array will have a heterogeneous dtype.

1 Like
10

Yeah sure, but only if the array is contiguous in memory, so you’d need to add a check first and handle the non-contiguous case.

1 Like
11

My bad, the function should be defined as vector_func. I made a change locally and then forgot to change it on this post (but the window has passed for me to edit the original post)

12

As I understand it, specifying the types makes the program faster because no work has to be done to determine the type of the argument and figure out which method to use. Does specifying the type as AbstractArray{<:Real} keep performance? I would think some work still has to be done to determine the types.

But I am new to Julia, so I could be misunderstanding the performance implications entirely.

13

Specifying types on functions doesn’t make them faster. Julia compiles specialized versions based on the types that are actually used.

14

Thank you for the very descriptive solution(s)!

Option 1 seems the most sensible. My main reason for avoiding that is because I am trying to create a python interface for someone else’s package, hopefully in a minimally invasive way. But I think that is what I will end up doing (but options 4 and 5 look like acceptable backups).