How to copy the result of python to Julia without allocation memory using PyCall

General Usage Performance
,
1

I have a Python program whose results will be saved in a big numpy array. I want to use this result in my Julia program, but I find that accessing the data will cause memory allocation in Julia. A demo is as follows:

using TimerOutputs
using PyCall

py"""
import numpy as np
def funa():
    return np.random.rand(36,36)
"""

function test_copy(B)
    @timeit "copy" B .= py"""funa()"""
end

B = zeros(36,36)
test_copy(B)

reset_timer!()
test_copy(B)
show(TimerOutputs.get_defaulttimer());

The output is:

 ────────────────────────────────────────────────────────────────────
                            Time                    Allocations      
                   ───────────────────────   ────────────────────────
 Tot / % measured:      191ΞΌs /  13.7%           16.1KiB /  71.7%    

 Section   ncalls     time    %tot     avg     alloc    %tot      avg
 ────────────────────────────────────────────────────────────────────
 copy           1   26.2ΞΌs  100.0%  26.2ΞΌs   11.6KiB  100.0%  11.6KiB
 ────────────────────────────────────────────────────────────────────

What I want to do is, since a numpy array has already been created in Python, how can I directly copy the data to Julia’s array A without causing memory allocation?

Thanks.

2

Does GitHub - mkitti/NumPyArrays.jl: Julia package to extend the conversion of Julia arrays to NumPy arrays help here?

3 Likes
3

Thank you for your reply.

If I do not misunderstand, NumPyArrays.jl is to convert Julia arrays into NumPy arrays in Python and the result of Python can be directly saved in those arrays.
However, what I want to do is access the array created by Python in Julia.

Thank you again.

4

Think about it for a minute. You can represent a Julia array as a NumPy array. Now you can try facilties such as numpy.copyto to move data.

From a low level perspective, we have two pointers and are trying to copy from one to the other.

I’m not exactly sure why the broadcasting call is allocating. It’s probably materializing the array.

You could also try grabbing the PyObject directly via py"..."o.

Note there is another package called PythonCall.jl which may perform differently here.

1 Like
5

I have found a way to achieve this goal, but I don’t know whether it is the most suitable way.

py"""
import numpy as np
def funa():
    A = np.random.rand(70,70)
    # print(A[0,0:6])
    return A
"""

function test_copy(B)
    @timeit "copy1" B .= py"""funa()"""
    @show B[1,1:6]
    # @timeit "copy2" begin
    #     A = py"""funa()"""o
    #     # @show A.__array_interface__
    #     ai = A.__array_interface__
    #     ip = ai["data"][1]
    #     @show ip
    #     Ap = Ptr{Float64}(ip)
    #     Bp = Base.unsafe_convert(Ptr{Float64}, B)
    #     unsafe_copyto!(Bp, Ap, 70*70)
    # end
    # @show B[1:6,1]
    @timeit "copy3" begin
        A = py"""funa()"""o   # get PyObject
        ip = py"""$A.__array_interface__["data"][0]"""
        # ip = py"""funa().__array_interface__["data"][0]"""   # Why is this result not correct???
        Ap = Ptr{Float64}(ip)
        Bp = Base.unsafe_convert(Ptr{Float64}, B)
        unsafe_copyto!(Bp, Ap, 70*70)
    end
    @show B[1:6,1]
end

B = zeros(70,70)
test_copy(B)

reset_timer!()
test_copy(B)
show(TimerOutputs.get_defaulttimer());

Result ( Only a little memory allocation):

 ────────────────────────────────────────────────────────────────────
                            Time                    Allocations      
                   ───────────────────────   ────────────────────────
 Tot / % measured:      259ΞΌs /  32.2%           52.5KiB /  77.1%    

 Section   ncalls     time    %tot     avg     alloc    %tot      avg
 ────────────────────────────────────────────────────────────────────
 copy1          1   47.3ΞΌs   56.6%  47.3ΞΌs   40.4KiB   99.7%  40.4KiB
 copy3          1   36.2ΞΌs   43.4%  36.2ΞΌs      112B    0.3%     112B
 ────────────────────────────────────────────────────────────────────
1 Like
6

That’s basically what needs to happen in the end. The game is to wrap the raw pointers into some type so that you only invoke unsafe_ when you actually know how many bytes to copy.

There is a probably a route involving PyCall.PyArray. Wrap that around a PyObject.

7

Oh yes, thanks. I will try it.

8

I believe it does, see in its docs:

Fast non-copying conversion of numeric arrays in either direction: modify Python arrays (e.g. bytes, array.array, numpy.ndarray) from Julia or Julia arrays from Python.

Note for PyCall:

Multidimensional arrays exploit the NumPy array interface for conversions between Python and Julia. By default, they are passed from Julia to Python without making a copy, but from Python to Julia a copy is made; no-copy conversion of Python to Julia arrays can be achieved with the PyArray type below.

Both package allow interopting with all Python libraries, and PyCall has fast(er) startup (and the py string macro, which I kind of miss, done differently in the alternative), other than that I believe PythonCall is the future, already many packages migrating to it from using PyCall (you CAN also use both together, according to the docs, if you do it right), because it has benefits, e.g. for dependency handling and:

Lossiness of conversion

Both packages allow conversion of Julia values to Python: PyObject(x) in PyCall, Py(x) in PythonCall.

Whereas both packages convert numbers, booleans, tuples and strings to their Python counterparts, they differ in handling other types. For example PyCall converts AbstractVector to list whereas PythonCall converts AbstractVector to juliacall.VectorValue which is a sequence type directly wrapping the Julia value - this has the advantage that mutating the Python object also mutates the original Julia object.
[…]
In fact, PythonCall has the policy that any mutable object will by default be wrapped in this way, which not only preserves mutability but makes conversion faster for large containers since it does not require taking a copy of all the data.
[…]
PyCall requires numpy to be installed, PythonCall doesn’t (it provides the same fast array access through the buffer protocol and array interface).

In its docs:

Helpful wrappers: interpret Python sequences, dictionaries, arrays, dataframes and IO streams as their Julia counterparts, and vice versa.

Dictionaries are by now ordered in Python, and while I would want them that way in Julia too (I tried to make a PR for that), they are not in Julia, so I’m not sure what happens with either package when converting. This could be a surprise.

Also Julia uses UTF-8 for strings, but Python doesn’t (though allows as a redundant representation), so it can’t be zero-copy when calling Python, until it changes to UTF-8 by default (and only) which I believe is planned. Neither in the other direction, though might be if UTF-8 also used, but I doubt that’s supported, but could be.

I don’t know how classes/object are handled…

9

Inspired by @mkitti , I have read the code of PyCall.jl and found another simple way to realize copying data with little memory allocation using PyCall as follows.

    @timeit "copy4" begin
        A = PyArray(py"""funa()"""o)
        copyto!(B, A)
    end
    @show B[end, end-5:end]

    Aa = PyArray(py"""funa()"""o)
    @timeit "copy5" begin
        Ao = py"""funa()"""o
        PyCall.setdata!(Aa, Ao)
        copyto!(B, Aa)
    end
    @show B[end, end-5:end]

Result:

 ────────────────────────────────────────────────────────────────────
                            Time                    Allocations      
                   ───────────────────────   ────────────────────────
 Tot / % measured:      655ΞΌs /  57.2%           62.3KiB /  66.3%    

 Section   ncalls     time    %tot     avg     alloc    %tot      avg
 ────────────────────────────────────────────────────────────────────
 copy1          1    113ΞΌs   30.3%   113ΞΌs   40.4KiB   97.8%  40.4KiB
 copy3          1    103ΞΌs   27.5%   103ΞΌs      112B    0.3%     112B
 copy4          1   82.6ΞΌs   22.1%  82.6ΞΌs      768B    1.8%     768B
 copy5          1   75.5ΞΌs   20.2%  75.5ΞΌs     32.0B    0.1%    32.0B
 ────────────────────────────────────────────────────────────────────

Thank @Palli very much for your detailed advice.
And I will try PythonCall if possible.