Calling Julia functions (which take custom structs as inputs) from Python

I’ve been trying to figure out a systematic way to call Julia functions from Python. I came across pyjulia library (in python) and PyCall modules (in julia) which seem to be pretty useful. However, most of the examples shown to demonstrate the usefulness of the packages, while extremely valuable and helpful, only consider inputs of type int, float, string, dict and arrays.
How can one call Julia functions that take other data types, say, custom Julia structs, as inputs? I’d really appreciate if somebody on this forum could offer some ideas/help on how I can call Julia functions with custom data structures other than the predefined types. I have provided an example below. The struct “Container” defined below is a simplified struct, and in a more general situation, could hold other custom structs. [I come from C++/Python background, and have heavily used Boost.Python and pybind11 to expose and extend Python to C++ and vice-versa. I am looking for something similar here]

#container.jl file:

export Container
export add_container_items
using LinearAlgebra

mutable struct Container{T<:Real}
x::T
y::T
function Container{T}(x::T, y::T) where {T}
return new(x, y)
end
function Container{T}(y::T) where {T}
return new(2, y)
end
end

function add_container_items(a::Container{T}) where{T}
y = a.x + a.y
return y
end

function add_numbers(x::Float64, y::Float64)
return x + y
end

Calling add_numbers() function from Python is straight forward, but how can I call add_container_items() function from Python, which takes a custom struct “container” as an input?

Thanks for any help in advance!

1 Like

What happens if you call the constructor of Container from Python?

One solution, if you cannot construct a Container on the Python side, is to create a Python interface that accepts types that pycall can handle.

@baggepinnen, thanks for your suggestion.

I have tried:

from julia import Main

Main.TestModules.Container(4.0, 5.0). <----- calling container constructor
(I’ve placed this “container” struct under a module named TestModules)

However, this throws an error:
JULIA: MethodError: no method matching Main.TestModules.Container(::Float64, ::Float64)
Stacktrace:
[1] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:790
[2] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:789
[3] _pyjlwrap_call(::Type, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /Users/apg/.julia/packages/PyCall/BcTLp/src/callback.jl:28
[4] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /Users/apg/.julia/packages/PyCall/BcTLp/src/callback.jl:49>

I think I kind of understand what you mean by creating a Python interface that accepts types that pycall can handle, but I am a bit lost on syntax (being a brand new Julia, PyCall and pyjulia user) I’d certainly appreciate any pointer.

Thanks a lot!

Have you imported the module on Python side (with PyJulia)?

In general, you can assign Python bindings to any Julia object and vice versa - in both languages, nearly everything is a first class object (incl. classes / structs, functions, etc.).

Consider the following Julia file “example.jl”

# example.jl
module ContainerTest

export Container
export add_container_items, add_numbers
using LinearAlgebra

mutable struct Container{T<:Real}
    x::T
    y::T
    function Container(x::T, y::T) where {T}
        return new{T}(x, y)
    end
    function Container(y::T) where {T}
        return new{T}(2, y)
    end
    function Container{T}(y::T) where {T}
        return new(3, y)
    end
end

function add_container_items(a::Container{T}) where{T}
    y = a.x + a.y
    return y
end

function add_numbers(x::Float64, y::Float64)
    return x + y
end

end # module

Take note, that I slightly changed the original definition of Container(y::T), because it is more usable, but for consistency added the original definition at the next step.

Now, in your python file, you can use this Julia file in the following way

from julia import Main
Main.include("example.jl")
Main.using(".ContainerTest")

add_numbers are straightforward:

Main.add_numbers(1.0, 2.0)
# 3.0

Simple container and add_container_items

c1 = Main.Container(1.0, 2.0)
Main.add_container_items(c1)
# 3.0

Container constructor with one argument

c2 = Main.Container(2)
Main.add_container_items(c2)
# 4

Container constructor with parametric type is more involved because there is no obvious python version, or at least I wasn’t able to find it. But you can use eval to build necessary variable

c3 = Main.eval("Container{Float64}(4.0)")
Main.add_container_items(c3)
# 7.0
4 Likes

@Skoffer, thank you so much for your detailed explanation ! That worked out nicely. I see the difference between your version and my version I was trying earlier. The template parameter “T” in the constructor was the reason why it was failing earlier. Moving that parameter to new{T} worked fine. As for the parametric constructor with {T}, looks like eval seems to be the way to go.

Now that this is at least working, I can try different scenarios where the Container member variables are other custom container/data structure other than predefined types like I had in the example above.

My only other question at this point is, how to properly manage the Main.include(“”) line in the code. For instance, in the Container example above (which is my custom library, say), let us assume that the custom struct and its functions make use of several core julia libraries/packages like Combinatorics, UniqueVectors, etc. to perform several mathematical operations for number crunching. The constructor and/or the function that I want to expose to Python will either take some custom structs (which I will expose to Python by calling the constructor) or other predefined types like array, numbers, dict, string, etc, which are already exposed to Python. They will never directly take any objects defined in core julia libraries. Inside the function body, which I want to expose to Python or call from Python, however, these functions could use one or many core Julia libraries to do the number crunching.

All of these other core julia packages are under my .julia/packages folder. Do I have to do Main.include(“path/to/core/julia/ package”) for each package used by the example above?

Is there a better way to accomplish this?

Currently, I am including these packages one-by-one. However, I keep getting errors due to the following lines:

Main.include( “/Users/apg/.julia/packages/RecipesBase/G4s6f/src/RecipesBase.jl” )

Main.include( “/Users/apg/.julia/packages/Polynomials/XpnvY/src/Polynomials.jl” )

Error:

ImportError: <PyCall.jlwrap (in a Julia function called from Python)

JULIA: LoadError: LoadError: ArgumentError: Package RecipesBase not found in current path:

Thanks a lot for your help! I really appreciate it.

Well, I think, that it can help if you try to run Julia code in Julia and only after everything compiles and runs correctly wrap it in python code. This way transition to the python environment should go much smoother since it adds a rather small wrapper around the usual Julia procedures.

I mean, you should treat your Julia scripts as usual Julia scripts, if you are using any libraries then they should be used in the usual way with using and other Module commands. You shouldn’t directly include library parts in python.

Consider the following example. In Julia’s example.jl you may write

using Roots
using Polynomials

function solve(a0, a1, a2, a3, x0)
    x = variable()
    poly = a0 + a1*x + a2*x^2 + a3*x^3
    find_zero(poly, x0)
end

Take the notion of using Roots and Polynomials at the beginning of the file. This is what is required by Julia’s code and Julia takes care of loading all the libraries properly.

In your python code you may use it as follows

from julia import Main
Main.include("example.jl")

Main.solve(1, 2, 3, 4, 0)
#  -0.605829586188268
4 Likes

@Skoffer, thanks a lot for your detailed explanation!

1 Like

@Skoffer, One thing I find a bit confusing about the approach you mentioned is that I can’t call “find_zero” function directly in Python even though that function is part of the well maintained Julia package and works in a Julia script. In the example above, you have a “wrapper” Julia script with a function “solve” defined locally. However, the Julia package Roots or Polynomials should already have a function called “find_zero” defined internally somewhere in the package. Is it not possible to expose those functions to Python without using a wrapper function?

from julia import Main
Main.include(“/path/to/Polynomials/and/Root”)

x0 = 0
a0 = 1
a1 = 2
a2 = 3
a3 = 4
x = Main.variable()
poly = a0 + a1x + a2x^2 + a3*x^3
Main.find_zero(poly, x0)

You do not need to include source files of packages ever. It should be always either using or import but never include.

Anyway, in the case of PyCall, there is a wrapper around using calls, so you can use Julia modules just by import them in python and use functions with fully qualified names. And, you can’t do poly = a0 + a1x + a2x^2 + a3*x^3 because python syntax does not support multiple dispatch, and +, ^ and so on are special operations defined for other types than numbers in this case. Actually, this is one of the reasons, why simple transpiling from python to julia is not working. But you still can use basic function calls and constructors.

from julia import Polynomials
from julia import Roots

poly = Polynomials.Polynomial([1, 2, 3, 4])
Roots.find_zero(poly, 0)
# -0.605829586188268

or, as usual in python, you can always import only relevant functions

from julia.Polynomials import Polynomial
from julia.Roots import find_zero
poly = Polynomial([1, 2, 3, 4])
find_zero(poly, 0)
# -0.605829586188268

Of course, Roots.jl and Polynomials.jl should be installed in Julia for this to work.

1 Like