mutable struct Person
age :: Int
weight :: Float64
end
function add_age!(p::Person)
p.age += 1
end
using PyCall
py_add_age = pyfunction(add_age!, Person)
Then, in Python,
p = Main.Person(10, 100.0)
Main.py_add_age(p)
This leads to the following conversion error:
JULIA: MethodError: Cannot `convert` an object of type PyObject to an object of type Person
How can I make this conversion possible? If I directly call the Julia function, it works, but just wanted to test the performance improvement via pyfunction.
I wouldn’t use pyfunction at all here. If you just do py_add_age = add_age!, then the default argument conversion should do what you want.
Otherwise you’d have to define a convert function, of the form:
function Base.convert(::Type{Person}, o::PyObject)
PyCall.is_pyjlwrap(o) || throw(ArgumentError("not a wrapped Julia object"))
return PyCall.unsafe_pyjlwrap_to_objref(o)::Person
end
Probably a slight benefit to using pyfunction and defining a convert function since it avoids a type-introspection step, but if you’re worried about the performance of as simple a function as add_age! from Python then you’re think about it all wrong — the Python overhead will swamp the cost of the call. pyfunction is more useful to customize which conversion is selected in cases where the default is not doing what you want.
To get a performance benefit from Julia in Python, you need to be calling Julia functions that do a lot of work, so that the overhead of calling them from Python is negligible. (It’s exactly analogous to calling a “vectorized” function, written in C or Fortran, from Numpy/Scipy — you “vectorize” so that the function has enough work to amortize the Python overhead.)
The above example was just a minimal example to illustrate what I wanted to do.
Does it mean that frequently calling of Julia functions from Python could not be so beneficial, depending on the job done in Julia?
The exact context is reinforcement learning, where pytorch trains neural networks where the simulator is done in Julia. Should it be the other way in this case? Julia calling Python, instead of Python calling Julia?
You can call Julia as frequently as you want, but to get a performance benefit the Julia function should be doing enough computational work on each call to make the Python-calling overhead negligible.
The exact context is reinforcement learning, where pytorch trains neural networks where the simulator is done in Julia. Should it be the other way in this case? Julia calling Python, instead of Python calling Julia?
Whether you have Julia calling Python or Python calling Julia, the overhead of Python is there. As long as your simulator is expensive enough, it shouldn’t matter (except for convenience/flexibility … I personally find it easier to work from Julia).
Where does “the overhead of Python” exactly come from? Handling Julia data in Python, or calling a Julia process from Python, or handling python data in Julia?
From the same reason why you can’t write fast inner-loop code in Python. Every object that is represented in Python, from an integer to a function, has to be wrapped in a heap-allocated wrapper (a PyObject) with a runtime type tag to indicate what type it, and every access to such an object is dynamic — you have to look at the type tag and dynamically dispatch to the correct code to extract the data and perform the operation you want.