# How to copy the result of python to Julia without allocation memory using PyCall

**URL:** <https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480>\
**Category:** Performance\
**Tags:** package, pycall\
**Created:** [December 12, 2023, 7:43am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480 "2023-12-12T07:43:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![wly](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wly/32/49517_2.png) [@wly](https://discourse.julialang.org/u/wly)\
**Post date:** [December 12, 2023, 7:43am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/1 "2023-12-12T07:43:24Z")

</div>

I have a Python program whose results will be saved in a big numpy array. I want to use this result in my Julia program, but I find that accessing the data will cause memory allocation in Julia. A demo is as follows:

```julia
using TimerOutputs
using PyCall

py"""
import numpy as np
def funa():
    return np.random.rand(36,36)
"""

function test_copy(B)
    @timeit "copy" B .= py"""funa()"""
end

B = zeros(36,36)
test_copy(B)

reset_timer!()
test_copy(B)
show(TimerOutputs.get_defaulttimer());

```

The output is:

```julia
 ────────────────────────────────────────────────────────────────────
                            Time Allocations      
                   ─────────────────────── ────────────────────────
 Tot / % measured: 191μs / 13.7% 16.1KiB / 71.7%    

 Section ncalls time %tot avg alloc %tot avg
 ────────────────────────────────────────────────────────────────────
 copy 1 26.2μs 100.0% 26.2μs 11.6KiB 100.0% 11.6KiB
 ────────────────────────────────────────────────────────────────────

```

What I want to do is, since a numpy array has already been created in Python, how can I directly copy the data to Julia’s array `A` without causing memory allocation?

Thanks.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [December 12, 2023, 7:52am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/2 "2023-12-12T07:52:41Z")

</div>

Does [GitHub - mkitti/NumPyArrays.jl: Julia package to extend the conversion of Julia arrays to NumPy arrays](https://github.com/mkitti/NumPyArrays.jl) help here?

---

<div class="post-metadata">

**Author:** ![wly](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wly/32/49517_2.png) [@wly](https://discourse.julialang.org/u/wly)\
**Post date:** [December 12, 2023, 8:14am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/3 "2023-12-12T08:14:14Z")

</div>

Thank you for your reply.

If I do not misunderstand, NumPyArrays.jl is to `convert Julia arrays into NumPy arrays in Python` and the result of Python can be directly saved in those arrays.  
However, what I want to do is access the array created by Python in Julia.

Thank you again.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [December 12, 2023, 9:26am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/4 "2023-12-12T09:26:35Z")

</div>

Think about it for a minute. You can represent a Julia array as a NumPy array. Now you can try facilties such as `numpy.copyto` to move data.

From a low level perspective, we have two pointers and are trying to copy from one to the other.

I’m not exactly sure why the broadcasting call is allocating. It’s probably materializing the array.

You could also try grabbing the `PyObject` directly via `py"..."o`.

Note there is another package called PythonCall.jl which may perform differently here.

---

<div class="post-metadata">

**Author:** ![wly](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wly/32/49517_2.png) [@wly](https://discourse.julialang.org/u/wly)\
**Post date:** [December 12, 2023, 9:26am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/5 "2023-12-12T09:26:46Z")

</div>

I have found a way to achieve this goal, but I don’t know whether it is the most suitable way.

```julia
py"""
import numpy as np
def funa():
    A = np.random.rand(70,70)
    # print(A[0,0:6])
    return A
"""

function test_copy(B)
    @timeit "copy1" B .= py"""funa()"""
    @show B[1,1:6]
    # @timeit "copy2" begin
    # A = py"""funa()"""o
    # # @show A. __array_interface__
    # ai = A. __array_interface__
    # ip = ai["data"][1]
    # @show ip
    # Ap = Ptr{Float64}(ip)
    # Bp = Base.unsafe_convert(Ptr{Float64}, B)
    # unsafe_copyto!(Bp, Ap, 70*70)
    # end
    # @show B[1:6,1]
    @timeit "copy3" begin
        A = py"""funa()"""o # get PyObject
        ip = py"""$A. __array_interface__ ["data"][0]"""
        # ip = py"""funa(). __array_interface__ ["data"][0]""" # Why is this result not correct???
        Ap = Ptr{Float64}(ip)
        Bp = Base.unsafe_convert(Ptr{Float64}, B)
        unsafe_copyto!(Bp, Ap, 70*70)
    end
    @show B[1:6,1]
end

B = zeros(70,70)
test_copy(B)

reset_timer!()
test_copy(B)
show(TimerOutputs.get_defaulttimer());

```

Result ( Only a little memory allocation):

```julia
 ────────────────────────────────────────────────────────────────────
                            Time Allocations      
                   ─────────────────────── ────────────────────────
 Tot / % measured: 259μs / 32.2% 52.5KiB / 77.1%    

 Section ncalls time %tot avg alloc %tot avg
 ────────────────────────────────────────────────────────────────────
 copy1 1 47.3μs 56.6% 47.3μs 40.4KiB 99.7% 40.4KiB
 copy3 1 36.2μs 43.4% 36.2μs 112B 0.3% 112B
 ────────────────────────────────────────────────────────────────────

```

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [December 12, 2023, 10:00am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/6 "2023-12-12T10:00:18Z")

</div>

That’s basically what needs to happen in the end. The game is to wrap the raw pointers into some type so that you only invoke `unsafe_` when you actually know how many bytes to copy.

There is a probably a route involving `PyCall.PyArray`. Wrap that around a `PyObject`.

---

<div class="post-metadata">

**Author:** ![wly](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wly/32/49517_2.png) [@wly](https://discourse.julialang.org/u/wly)\
**Post date:** [December 12, 2023, 10:00am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/7 "2023-12-12T10:00:49Z")

</div>

Oh yes, thanks. I will try it.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [December 12, 2023, 11:16am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/8 "2023-12-12T11:16:45Z")

</div>

> [@mkitti](#):
>
> Note there is another package called [PythonCall.jl](https://juliahub.com/ui/Packages/PythonCall) which may perform differently here.

I believe it does, see in its docs:

> Fast non-copying conversion of numeric arrays in either direction: modify Python arrays (e.g. `bytes`, `array.array`, `numpy.ndarray`) from Julia or Julia arrays from Python.

Note for PyCall:

> Multidimensional arrays exploit the NumPy array interface for conversions between Python and Julia. By default, they are passed from Julia to Python without making a copy, but from Python to Julia a copy is made; no-copy conversion of Python to Julia arrays can be achieved with the `PyArray` type below.

Both package allow interopting with all Python libraries, and PyCall has fast(er) startup (and the py string macro, which I kind of miss, done differently in the alternative), other than that I believe PythonCall is the future, already many packages migrating to it from using PyCall (you CAN also use both together, according to the docs, if you do it right), because it has benefits, e.g. for dependency handling and:

> ### [Lossiness of conversion](https://juliapy.github.io/PythonCall.jl/stable/pycall/#Lossiness-of-conversion)
> 
> Both packages allow conversion of Julia values to Python: `PyObject(x)` in PyCall, `Py(x)` in PythonCall.
> 
> Whereas both packages convert numbers, booleans, tuples and strings to their Python counterparts, they differ in handling other types. For example PyCall converts `AbstractVector` to `list` whereas PythonCall converts `AbstractVector` to `juliacall.VectorValue` which is a sequence type directly wrapping the Julia value - this has the advantage that mutating the Python object also mutates the original Julia object.  
> […]  
> In fact, PythonCall has the policy that any mutable object will by default be wrapped in this way, which not only preserves mutability but makes conversion faster for large containers since it does not require taking a copy of all the data.  
> […]  
> PyCall requires numpy to be installed, PythonCall doesn’t (it provides the same fast array access through the buffer protocol and array interface).

In its docs:

> Helpful wrappers: interpret Python sequences, dictionaries, arrays, dataframes and IO streams as their Julia counterparts, and vice versa.

Dictionaries are by now ordered in Python, and while I would want them that way in Julia too (I tried to make a PR for that), they are not in Julia, so I’m not sure what happens with either package when converting. This could be a surprise.

Also Julia uses UTF-8 for strings, but Python doesn’t (though allows as a redundant representation), so it can’t be zero-copy when calling Python, until it changes to UTF-8 by default (and only) which I believe is planned. Neither in the other direction, though might be if UTF-8 also used, but I doubt that’s supported, but could be.

I don’t know how classes/object are handled…

---

<div class="post-metadata">

**Author:** ![wly](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wly/32/49517_2.png) [@wly](https://discourse.julialang.org/u/wly)\
**Post date:** [December 12, 2023, 11:49am UTC](https://discourse.julialang.org/t/how-to-copy-the-result-of-python-to-julia-without-allocation-memory-using-pycall/107480/9 "2023-12-12T11:49:19Z")

</div>

Inspired by @mkitti , I have read the code of PyCall.jl and found another simple way to realize copying data with little memory allocation using PyCall as follows.

```julia
    @timeit "copy4" begin
        A = PyArray(py"""funa()"""o)
        copyto!(B, A)
    end
    @show B[end, end-5:end]

    Aa = PyArray(py"""funa()"""o)
    @timeit "copy5" begin
        Ao = py"""funa()"""o
        PyCall.setdata!(Aa, Ao)
        copyto!(B, Aa)
    end
    @show B[end, end-5:end]

```

Result:

```julia
 ────────────────────────────────────────────────────────────────────
                            Time Allocations      
                   ─────────────────────── ────────────────────────
 Tot / % measured: 655μs / 57.2% 62.3KiB / 66.3%    

 Section ncalls time %tot avg alloc %tot avg
 ────────────────────────────────────────────────────────────────────
 copy1 1 113μs 30.3% 113μs 40.4KiB 97.8% 40.4KiB
 copy3 1 103μs 27.5% 103μs 112B 0.3% 112B
 copy4 1 82.6μs 22.1% 82.6μs 768B 1.8% 768B
 copy5 1 75.5μs 20.2% 75.5μs 32.0B 0.1% 32.0B
 ────────────────────────────────────────────────────────────────────

```

Thank @Palli very much for your detailed advice.  
And I will try `PythonCall` if possible.