JuliaCall and SimpleChains not working together

General Usage
,
1

Hi,
I have been working on a surrogate model, based on SimpleChains NNs, and I am quite satisfied with the performance.

from juliacall import Main as jl

jl.seval(“using Capse”)
jl.seval(“using SimpleChains”)

Since I need to deliver the trained surrogate models to python people, I have been trying to build a small python wrapper and I decided to use JuliaCall. It looks very nice, but I am not being able to run SImpleChains from the python interface.

Here I’ll write a small (quick and dirty) MWE just to show my problem

import numpy as np
from juliacall import Main as jl

jl.seval("using SimpleChains")
mlpd = jl.seval('SimpleChain(static(8), TurboDense(tanh, 64), TurboDense(tanh, 64), TurboDense(tanh, 64), TurboDense(identity, 4))')
p = jl.seval('SimpleChains.init_params(SimpleChain(static(8), TurboDense(tanh, 64), TurboDense(tanh, 64), TurboDense(tanh, 64),TurboDense(identity, 4)))')

Now, if I run

input_test = np.random.rand(8)
mlpd(input_test,p)

I get the following error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mbonici/.julia/packages/PythonCall/2Y5CR/src/jlwrap/any.jl", line 201, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
juliacall.JuliaError: "Memory access for PythonCall.PyArray{Float64, 1, true, true, Float64} not implemented yet."
Stacktrace:
  [1] memory_reference
    @ ~/.julia/packages/LayoutPointers/ssuuk/src/stridedpointers.jl:59 [inlined]
  [2] memory_reference
    @ ~/.julia/packages/LayoutPointers/ssuuk/src/stridedpointers.jl:16 [inlined]
  [3] stridedpointer
    @ ~/.julia/packages/LayoutPointers/ssuuk/src/stridedpointers.jl:93 [inlined]
  [4] PtrArray
    @ ~/.julia/packages/StrideArraysCore/VQxXL/src/ptr_array.jl:33 [inlined]
  [5] TurboDense
    @ ~/.julia/packages/SimpleChains/HhLUa/src/dense.jl:150 [inlined]
  [6] __chain
    @ ~/.julia/packages/SimpleChains/HhLUa/src/simple_chain.jl:286 [inlined]
  [7] _chain
    @ ~/.julia/packages/SimpleChains/HhLUa/src/simple_chain.jl:336 [inlined]
  [8] with_heap_memory(::typeof(SimpleChains._chain), ::SimpleChain{Tuple{StaticInt{8}}, Tuple{TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{2499}, typeof(identity)}}}, ::StaticInt{22623}, ::PythonCall.PyArray{Float64, 1, true, true, Float64}, ::Ptr{Float32})
    @ SimpleChains ~/.julia/packages/SimpleChains/HhLUa/src/memory.jl:36
  [9] (::SimpleChain{Tuple{StaticInt{8}}, Tuple{TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{2499}, typeof(identity)}}})(arg::PythonCall.PyArray{Float64, 1, true, true, Float64}, params::StrideArraysCore.StaticStrideArray{Tuple{StaticInt{179651}}, (true,), Float32, 1, 1, 0, (1,), Tuple{StaticInt{4}}, Tuple{StaticInt{1}}, 179651})
    @ SimpleChains ~/.julia/packages/SimpleChains/HhLUa/src/simple_chain.jl:170
 [10] pyjlany_call(self::SimpleChain{Tuple{StaticInt{8}}, Tuple{TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{64}, typeof(tanh)}, TurboDense{true, StaticInt{2499}, typeof(identity)}}}, args_::PythonCall.Py, kwargs_::PythonCall.Py)
    @ PythonCall ~/.julia/packages/PythonCall/2Y5CR/src/jlwrap/any.jl:31
 [11] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
    @ PythonCall ~/.julia/packages/PythonCall/2Y5CR/src/jlwrap/base.jl:69
 [12] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
    @ PythonCall.C ~/.julia/packages/PythonCall/2Y5CR/src/cpython/jlwrap.jl:47

I have already built a small wrapper using JuliaCall and never found such an error.
Thank you for your help with this issue.
(Tagging @ChrisRackauckas @Elrod and @cjdoris )
Cheers,
Marco
Edit: Ialready looked into pywrap/PyArray.jl, after reading the error, but I don’t think I would be able to modify it and/or make a PR.

1 Like
2

Would copying be a solution? I ran into a similar issue at some point and solved the problem on the julia side with

function_to_call_from_python(input, p) = mlpd(copy(input), p)
1 Like
3

Now I am having lunch.
But sure, I’ll try it as soon as I get access to my computer again.
Thanks for the suggestion!

4

I just tried and it didn’t work :sweat: .
The error message I obtain is

juliacall.JuliaError: MethodError: no method matching copy(::PythonCall.Py)
5

Something like this should work:

import numpy as np
from juliacall import Main as jl

jl.seval("using SimpleChains")
mlpd = jl.seval('SimpleChain(static(8), TurboDense(tanh, 64), TurboDense(tanh, 64), TurboDense(tanh, 64), TurboDense(identity, 4))')
p = jl.SimpleChains.init_params(mlpd)

input_test = np.random.rand(8)
mlpd(jl.collect(input_test), p)

What’s going on is that when the numpy array is passed to a Julia function, it is automatically converted to a PyArray (which wraps the underlying data as an AbstractArray). From the error you posted, it appears that your model requires a strided array as input. Now PyArray is a strided array, but ArrayInterface.jl does not know this, hence the error.

The above code calls collect on the PyArray, which will convert it to a normal Array, which should work in your model.

Alternatively you can convert straight to Array (skipping the intermediate PyArray) like so:

from juliacall import convert as jlconvert
mlpd(jlconvert(jl.Array, input_test), p)

This way is more verbose, but also more explicit about what conversion is happening.

The real solution is for PythonCall to define the right overloads for ArrayInterface so that it knows PyArray is strided. There’s an issue for it here: Use ArrayInterface.jl? · Issue #61 · cjdoris/PythonCall.jl · GitHub.

2 Likes
6

Thank you!
I tried what you suggested and it worked…thank you!

7

Alternatively, it should be possible to support the ArrayInterface with a PyArray.

8

Yep thanks I noted that at the end of my reply.

9

Ah, yes, and an issue you made from October 2021. You’re way ahead of me. :wink:

1 Like