How to Efficiently Index PyArrays?

General Usage Performance
, ,
21

The argument is pyarray not array. So you are timing reading a global variable and not touching PyCall at all.

1 Like
22

Embarrassing mistake on my side. Corrected test:

using PyCall

function index_test(pyarray::PyArray{UInt8, 3})
    s = 0
    for i in 1:size(pyarray, 1)
        for j in 1:size(pyarray, 2)
            for k in 1:size(pyarray, 3)
                s += pyarray[i, j, k]
            end
        end
    end
    s
end

function from_python(pyarray::PyArray{UInt8, 3})
    return convert(Array{UInt8, 3}, pyarray)
end

function index_test(array::Array{UInt8, 3})
    s = 0
    for i in 1:size(array, 1)
        for j in 1:size(array, 2)
            for k in 1:size(array, 3)
                s += array[i, j, k]
            end
        end
    end
    s
end

function to_python(array::Array{UInt8, 3})
    PyArray(PyObject(array))
end

array = Array{UInt8, 3}(undef, (250, 250, 3))
@btime index_test($array)
@btime to_python($array)
pyarray = PyArray(PyObject(array))
@btime index_test($pyarray)
@btime from_python($pyarray)
println()

with result

  87.800 μs (0 allocations: 0 bytes)
  1.820 μs (20 allocations: 1.16 KiB)
  161.300 μs (0 allocations: 0 bytes)
  469.700 μs (2 allocations: 183.20 KiB)

Everything looks fine!

1 Like
23

My bad. Tested it again. It is 50us faster with the Array{Float32, 2}(undef, 92, 120). I originally tested it with Matrix{Float32}(undef, 92, 120).

24

those are exactly the same

25

I am new to Julia so sorry if I seem so naive.

26

I tried the index test with different versions of PyCall. Using v1.90.0, only 11 allocations are made. Using any earlier version gives me the error:

ERROR: LoadError: InitError: could not load symbol "Py_CompileString":
The specified procedure could not be found. 
Stacktrace:
 [1] macro expansion
   @ C:\Users\user\.julia\packages\PyCall\WcrLS\src\exception.jl:81 [inlined]
 [2] pyeval_(s::String, globals::PyCall.PyDict{String, PyCall.PyObject, false}, locals::PyCall.PyDict{String, PyCall.PyObject, false}, input_type::Int64, fname::String)
   @ PyCall C:\Users\user\.julia\packages\PyCall\WcrLS\src\pyeval.jl:20
 [3] pyeval_
   @ C:\Users\user\.julia\packages\PyCall\WcrLS\src\pyeval.jl:17 [inlined]
 [4] __init__()
   @ PyCall C:\Users\user\.julia\packages\PyCall\WcrLS\src\pyinit.jl:131
 [5] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base .\loading.jl:696
 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base .\loading.jl:782
 [7] _require(pkg::Base.PkgId)
   @ Base .\loading.jl:1020
 [8] require(uuidkey::Base.PkgId)
   @ Base .\loading.jl:936
 [9] require(into::Module, mod::Symbol)
   @ Base .\loading.jl:923
during initialization of module PyCall
in expression starting at C:\Users\user\PycharmProjects\juliaProject\arcade.jl:7