Creating an package (and eventually app) with Python dependencies

I’m working on my first package ever in any language. I want to eventually turn this package into an standalone app (using PackageCompiler.jl). I am running into some trouble.

My (local) package MyPackage uses a python package PyPackage. I’ve pip-installed PyPackage using Conda.jl, and am using PyCall.jl to use it. I use PyPackage functionality to define my own struct S, and later to do some processing on instances of S. The contents of MyPackage/src/MyPackage.jl look something like what I’ve shown below.

When I run this file alone, all goes well, and I am able to define instances of S and use my_func.

When I do using MyPackage, I get errors. For instance, when I do S(), I get a complaint from somewhere in .julia/packages/PyCall that py_f() is not defined.

I believe this is an issue with how my environment is setup, and possibly a limitation of PyCall.

I suspect that when I eventually turn the package into an app, I will need Python, PyCall, MyPackage, and PyPackage all to be working in/from the same environment. But Conda.jl’s documentation states that “If you are installing Python packages for use with PyCall, you must use the root environment.”

Currently, PyCall seems to be running from .julia/packages, even though I have added it to MyPackage’s Project.toml as a dependency.

My questions:

  1. Is my understanding correct that Python, PyCall, MyPackage, and PyPackage all will need to be working in/from the same environment for me to ultimately turn MyPackage into an app?

  2. If yes, does it mean that PyCall.jl + Conda.jl combination can not achieve this? I prefer PyCall because of the py"" syntax.

  3. If I have to switch to PythonCall.jl + CondaPkg.jl, how would I re-do this? (If you could kindly show me by re-work the toy code below.)

(If you respond, just know that I might need to come back to this post tomorrow.)

module MyPackage

ENV["PythonPackage_API_KEY"] = "xxx" # need to set before `using PyCall`

using PyCall

jl_func(x) = # a function that does some processing 

py"""
from PyPackage import py_f, py_g, py_h
"""

struct S
    a

    function S()
        a = jl_func(py"py_f()")     # -----> python

        return new(a)
    end
end

function my_func(s::S)
    py"""
    x = py_g($s.a)                  # -----> python
    """

    result = py"py_h(x)"            # -----> python

    return result
end

end # end of module

I usually recommend PythonCall by now (packages are increasingly migrating to it, it’s the future, well present of Py-interop), for e.g. dependency handling, and better API. py"" syntax which I liked, and it’s the only thing I miss, plus the fast startup of PyCall.

In case it helps you can use together (then read carefully):

Then you could use PythonCall as suggested, and the other syntax just as an exception, or well often (or always), then maybe no point in having both.

For a package that others use I think PythonCall would be much better because of the dependencies. It would even download (latest, or asked for) Python automatically for users. For an app however it may NOT be better.

It seems good that your app can be smaller, not bundling Python or Python dependencies, only references to, and then auto-downloaded on first use.

But that requires an internet connection (that most have…), enabled. I fully support such Julia apps, as a default, they could even download Julia and Julia package dependencies (and a non-default option bundling all if you’re paranoid about internet not working).

If you really want to bundle everything then PyCall may be better, but then you need to figure out where all the Python stuff goes and package it all up manually. It’s likely not difficult, though I’m not sure. It might go into a different folder, and you want to redirect it to your Julia code folder.

If the main reason to use PythonCall.jl is its dependency CondaPkg.jl, then maybe you can use just that and PyCall?

@cjdoris

In general, PythonCall and JuliaCall are only supported on platforms with Tier 1 level of support by Julia. Currently, Apple silicon is Tier 2, so is not supported.

Note, that doc is outdated Apple Silicon is now tier 1. I’m still not sure though if PythonCall supports it. Likely PythonCall doesn’t care, or is no less compatible than PyCall. It would be a question do the Python dependencies do.

Thanks @Palli . I did come across the documentation you share. And actually, while the documentation says that PyCall and PythonCall can both be used together, my Julia session crashes if I do using PyCall; using PythonCall (in either order). I’m on a mac with Julia 1.9. I did do pushfirst!(pyimport("sys")."path", "") at one point as mentioned in the PyCall documentation. I’m not sure that has anything to do with it.

I’m also going to tag @stevengj to see if they have the time to comment. I can’t be the first or last person to wan to make a Julia app with a python dependency.

I haven’t used together myself, just know about the issue and solution in the doc (why I wrote read carefully, didn’t it work?):

  • To force PythonCall to use the same Python interpreter as PyCall, set the environment variable JULIA_PYTHONCALL_EXE to "@PyCall". Note that this will opt out of automatic dependency management using CondaPkg.
  • Alternatively, to force PyCall to use the same interpreter as PythonCall, set the environment variable PYTHON to PythonCall.C.CTX.exe_path and then Pkg.build("PyCall"). You will need to do this each time you change project, because PythonCall by default uses a different Python for each project.
1 Like

I had seen that. I didn’t know that the error could be due to PyCall and PythonCall trying to run different python interpreters. In any case, I might try a few more things before trying to make PyCall and PythonCall work together, particularly as I don’t see if/how it would solve my problem.

There’ s also jnumpy/TyPython at main · Suzhou-Tongyuan/jnumpy (github.com) which has building portable sysimg in mind. I haven’ t used it directly but jnumpy is nice.

2 Likes

I edited the response below after @fgerick pointed out that the solution is in the documentation.

==============

I think I figured it out.

I switched to PythonCall and CondaPkg, but ran into the same problem. The solution is here, as pointed out by @fgerick :

It’s something to do with pointers. I don’t understand pointers (they are things that point to places in physical memory). Basically if you precompile a module that is using python under the hood, python’s pointers get messed up. Apparently the solution is to create references to those Pythonland things in Julialand and then update the contents of the Julialand references AFTER the module has loaded, inside the module’s __init__ function. PyCall has PyNULL you can use for it (as shown in the documentation). For PythonCall, Refs seem to work.

So the module needs to look something like this, it should work if you import it as a module using .MadModule or as a package using MadModule (after the necessary package-related setup).

module MadModule

export julia_function_using_python_function

using PythonCall

const python_function = Ref{Py}() # an empty Ref that can hold PythonCall.Py objects

function julia_function_using_python_function()
	
	result_from_python_package = python_function[]() # notice the []

	result = # more calculation on result_from_python_package

	return result
end

function __init__()
    python_function[] = pyimport("PythonPackage").some_function
end

end
2 Likes

I think it’s already there: Guide · PythonCall & JuliaCall

1 Like

Oh amazing. And amazingly frustrating that I completely missed that. I think I was doing something very wrong when navigating the docs. Will update my response above.

I fully support you using Julia (as your main or only language). Since you use Python with, also great, you must decide, if you call Python from Julia, as you decided, or Python is the main language, calling Julia.

Both are supported with the PythonCall repo I’ve already mentioned.

But since TyPython was brought up I point to, see also rather new (I think neither yet announced) intriguing project if you want to use Julia as a “C” extension (to e.g Python, as of last month Support Python Bindings):

TyJuliaCAPI.jl provides a stable C API for Julia.

These APIs are provided in the form of function pointers, and there are two initialization methods:

  1. If the caller is Julia, […]
  2. If the caller is an external language, […] In this case, we can enable a network protocol to transmit function pointers over the network, allowing both two side to access each other’s C API (this approach is inspired by MATLAB’s interoperability).

Motivation
The need for TyJuliaCAPI arises from a technical challenge: How can Julia interact with other external languages within the same process?

We have [several] projects implemented in C++/C#, and calling Julia from native code is a crucial requirement.

So, how to call Julia from C#: The only solution to this problem is to use the C API.

However, although Julia provides a C API (julia.h), these APIs have the following issues: […]

  1. The API is unstable and changes with every version. […]

And we need a stable and generic Julia C API.

In the past, we have learned a useful technique from PythonCall.jl (referred to as GC Pooling), which PythonCall itself used to provide a set of interoperation mechanisms, surpassing similar projects in terms of stability and performance. Following this technique, we have implemented a stable and well-designed C API for Julia, which we call TyJuliaCAPI.

In TyJuliaCAPI, the lifecycle management is adopted in a manner similar to the Python Stable C API (the most widely applied and highly regarded C API design)
[…]
The complete API list can be found in include/tyjuliacapi.hpp.

External Language Binding

Currently, we have implemented a C# language binding generator, which can be found in bindings/csharp.jl