# Creating an package (and eventually app) with Python dependencies

**URL:** <https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317>\
**Category:** New to Julia\
**Tags:** package, pycall, pythoncall\
**Created:** [December 8, 2023, 6:03am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317 "2023-12-08T06:03:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sadish-d](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sadish-d/32/48058_2.png) [@sadish-d](https://discourse.julialang.org/u/sadish-d)\
**Post date:** [December 8, 2023, 6:03am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/1 "2023-12-08T06:03:22Z")

</div>

I’m working on my first package ever in any language. I want to eventually turn this package into an standalone app (using PackageCompiler.jl). I am running into some trouble.

My (local) package `MyPackage` uses a python package `PyPackage`. I’ve pip-installed `PyPackage` using `Conda.jl`, and am using `PyCall.jl` to use it. I use `PyPackage` functionality to define my own struct `S`, and later to do some processing on instances of `S`. The contents of `MyPackage/src/MyPackage.jl` look something like what I’ve shown below.

When I run this file alone, all goes well, and I am able to define instances of `S` and use `my_func`.

When I do `using MyPackage`, I get errors. For instance, when I do `S()`, I get a complaint from somewhere in `.julia/packages/PyCall` that `py_f()` is not defined.

I believe this is an issue with how my environment is setup, and possibly a limitation of PyCall.

I suspect that when I eventually turn the package into an app, I will need Python, PyCall, MyPackage, and PyPackage all to be working in/from the same environment. But `Conda.jl`’s documentation states that “If you are installing Python packages for use with PyCall, you must use the root environment.”

Currently, PyCall seems to be running from `.julia/packages`, even though I have added it to MyPackage’s Project.toml as a dependency.

My questions:

1. Is my understanding correct that Python, PyCall, MyPackage, and PyPackage all will need to be working in/from the same environment for me to ultimately turn MyPackage into an app?

2. If yes, does it mean that PyCall.jl + Conda.jl combination can not achieve this? I prefer PyCall because of the `py""` syntax.

3. If I have to switch to PythonCall.jl + CondaPkg.jl, how would I re-do this? (If you could kindly show me by re-work the toy code below.)

(If you respond, just know that I might need to come back to this post tomorrow.)

```julia
module MyPackage

ENV["PythonPackage_API_KEY"] = "xxx" # need to set before `using PyCall`

using PyCall

jl_func(x) = # a function that does some processing 

py"""
from PyPackage import py_f, py_g, py_h
"""

struct S
    a

    function S()
        a = jl_func(py"py_f()") # -----> python

        return new(a)
    end
end

function my_func(s::S)
    py"""
    x = py_g($s.a) # -----> python
    """

    result = py"py_h(x)" # -----> python

    return result
end

end # end of module

```

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [December 8, 2023, 12:00pm UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/2 "2023-12-08T12:00:01Z")

</div>

> **[Coming from PyCall/PyJulia? · PythonCall & JuliaCall](https://juliapy.github.io/PythonCall.jl/stable/pycall/)**
>
> Documentation for PythonCall & JuliaCall.

> [@sadish-d](#):
>
> - I prefer PyCall because of the `py""` syntax.
> - If I have to switch to [PythonCall.jl](https://juliahub.com/ui/Packages/PythonCall) + [CondaPkg.jl](https://juliahub.com/ui/Packages/CondaPkg), how would I re-do this?

I usually recommend PythonCall by now (packages are increasingly migrating to it, it’s the future, well present of Py-interop), for e.g. dependency handling, and better API. `py""` syntax which I liked, and it’s the only thing I miss, plus the fast startup of PyCall.

In case it helps you can use together (then read carefully):

> **[Coming from PyCall/PyJulia? · PythonCall & JuliaCall](https://juliapy.github.io/PythonCall.jl/stable/pycall/)**
>
> Documentation for PythonCall & JuliaCall.

Then you could use PythonCall as suggested, and the other syntax just as an exception, or well often (or always), then maybe no point in having both.

For a package that others use I think PythonCall would be much better because of the dependencies. It would even download (latest, or asked for) Python automatically for users. For an app however it may NOT be better.

It seems good that your app can be smaller, not bundling Python or Python dependencies, only references to, and then auto-downloaded on first use.

But that requires an internet connection (that most have…), enabled. I fully support such Julia apps, as a default, they could even download Julia and Julia package dependencies (and a non-default option bundling all if you’re paranoid about internet not working).

If you really want to bundle everything then PyCall may be better, but then you need to figure out where all the Python stuff goes and package it all up manually. It’s likely not difficult, though I’m not sure. It might go into a different folder, and you want to redirect it to your Julia code folder.

If the main reason to use [PythonCall.jl](https://juliahub.com/ui/Packages/PythonCall) is its dependency [CondaPkg.jl](https://juliahub.com/ui/Packages/CondaPkg), then maybe you can use just that and PyCall?

@cjdoris

> In general, PythonCall and JuliaCall are only supported on platforms with [Tier 1](https://julialang.org/downloads/#supported_platforms) level of support by Julia. Currently, Apple silicon is Tier 2, so is not supported.

Note, that doc is outdated Apple Silicon is now tier 1. I’m still not sure though if PythonCall supports it. Likely PythonCall doesn’t care, or is no less compatible than PyCall. It would be a question do the Python dependencies do.

---

<div class="post-metadata">

**Author:** ![sadish-d](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sadish-d/32/48058_2.png) [@sadish-d](https://discourse.julialang.org/u/sadish-d)\
**Post date:** [December 8, 2023, 3:10pm UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/3 "2023-12-08T15:10:06Z")

</div>

Thanks @Palli . I did come across the documentation you share. And actually, while the documentation says that PyCall and PythonCall can both be used together, my Julia session crashes if I do `using PyCall; using PythonCall` (in either order). I’m on a mac with Julia 1.9. I did do `pushfirst!(pyimport("sys")."path", "")` at one point as mentioned in the PyCall documentation. I’m not sure that has anything to do with it.

I’m also going to tag @stevengj to see if they have the time to comment. I can’t be the first or last person to wan to make a Julia app with a python dependency.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [December 8, 2023, 4:34pm UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/4 "2023-12-08T16:34:20Z")

</div>

I haven’t used together myself, just know about the issue and solution in the doc (why I wrote read carefully, didn’t it work?):

> - To force PythonCall to use the same Python interpreter as PyCall, set the environment variable `JULIA_PYTHONCALL_EXE` to `"@PyCall"`. Note that this will opt out of automatic dependency management using CondaPkg.
> - Alternatively, to force PyCall to use the same interpreter as PythonCall, set the environment variable `PYTHON` to `PythonCall.C.CTX.exe_path` and then `Pkg.build("PyCall")`. You will need to do this each time you change project, because PythonCall by default uses a different Python for each project.

---

<div class="post-metadata">

**Author:** ![sadish-d](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sadish-d/32/48058_2.png) [@sadish-d](https://discourse.julialang.org/u/sadish-d)\
**Post date:** [December 8, 2023, 7:41pm UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/5 "2023-12-08T19:41:01Z")

</div>

I had seen that. I didn’t know that the error could be due to PyCall and PythonCall trying to run different python interpreters. In any case, I might try a few more things before trying to make PyCall and PythonCall work together, particularly as I don’t see if/how it would solve my problem.

---

<div class="post-metadata">

**Author:** ![xgdgsc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xgdgsc/32/608_2.png) [@xgdgsc](https://discourse.julialang.org/u/xgdgsc)\
**Post date:** [December 9, 2023, 12:57am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/6 "2023-12-09T00:57:18Z")

</div>

There’ s also [jnumpy/TyPython at main · Suzhou-Tongyuan/jnumpy (github.com)](https://github.com/Suzhou-Tongyuan/jnumpy/tree/main/TyPython) which has building portable sysimg in mind. I haven’ t used it directly but jnumpy is nice.

---

<div class="post-metadata">

**Author:** ![sadish-d](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sadish-d/32/48058_2.png) [@sadish-d](https://discourse.julialang.org/u/sadish-d)\
**Post date:** [December 9, 2023, 4:42am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/7 "2023-12-09T04:42:20Z")

</div>

I edited the response below after @fgerick pointed out that the solution is in the documentation.

==============

I think I figured it out.

I switched to PythonCall and CondaPkg, but ran into the same problem. The solution is here, as pointed out by @fgerick :

> **[Guide · PythonCall & JuliaCall](https://juliapy.github.io/PythonCall.jl/stable/pythoncall/#Precompilation)**
>
> Documentation for PythonCall & JuliaCall.

It’s something to do with pointers. I don’t understand pointers (they are things that point to places in physical memory). Basically if you precompile a module that is using python under the hood, python’s pointers get messed up. Apparently the solution is to create references to those Pythonland things in Julialand and then update the contents of the Julialand references AFTER the module has loaded, inside the module’s ` __init__ ` function. PyCall has `PyNULL` you can use for it (as shown in the documentation). For PythonCall, `Ref`s seem to work.

So the module needs to look something like this, it should work if you import it as a module `using .MadModule` or as a package `using MadModule` (after the necessary package-related setup).

```julia
module MadModule

export julia_function_using_python_function

using PythonCall

const python_function = Ref{Py}() # an empty Ref that can hold PythonCall.Py objects

function julia_function_using_python_function()
	
	result_from_python_package = python_function[]() # notice the []

	result = # more calculation on result_from_python_package

	return result
end

function __init__ ()
    python_function[] = pyimport("PythonPackage").some_function
end

end

```

---

<div class="post-metadata">

**Author:** ![fgerick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fgerick/32/13228_2.png) [@fgerick](https://discourse.julialang.org/u/fgerick)\
**Post date:** [December 9, 2023, 5:14am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/8 "2023-12-09T05:14:42Z")

</div>

> [@sadish-d](#):
>
> I’m going to mark this as a solution for now, but would be nice if someone more knowledgable can confirm this. And would definitely be nice to add this to PythonCall’s documentation.

I think it’s already there: [Guide · PythonCall & JuliaCall](https://juliapy.github.io/PythonCall.jl/stable/pythoncall/#Precompilation)

---

<div class="post-metadata">

**Author:** ![sadish-d](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sadish-d/32/48058_2.png) [@sadish-d](https://discourse.julialang.org/u/sadish-d)\
**Post date:** [December 9, 2023, 5:30am UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/9 "2023-12-09T05:30:28Z")

</div>

Oh amazing. And amazingly frustrating that I completely missed that. I think I was doing something very wrong when navigating the docs. Will update my response above.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [December 9, 2023, 6:37pm UTC](https://discourse.julialang.org/t/creating-an-package-and-eventually-app-with-python-dependencies/107317/10 "2023-12-09T18:37:21Z")

</div>

I fully support you using Julia (as your main or only language). Since you use Python with, also great, you must decide, if you call Python from Julia, as you decided, or Python is the main language, calling Julia.

Both are supported with the PythonCall repo I’ve already mentioned.

But since TyPython was brought up I point to, see also rather new (I think neither yet announced) intriguing project if you want to use Julia as a “C” extension (to e.g Python, as of last month [Support Python Bindings](https://github.com/Suzhou-Tongyuan/TyJuliaCAPI.jl/commit/be07e4dcf0a9f0a2bdff71f00dd4d82b2f3f8687)):

> **[GitHub - Suzhou-Tongyuan/TyJuliaCAPI.jl: Stable and generic C API for Julia](https://github.com/Suzhou-Tongyuan/TyJuliaCAPI.jl)**
>
> Stable and generic C API for Julia. Contribute to Suzhou-Tongyuan/TyJuliaCAPI.jl development by creating an account on GitHub.

> TyJuliaCAPI.jl provides a stable C API for Julia.
> 
> These APIs are provided in the form of function pointers, and there are two initialization methods:
> 
> 1. If the caller is Julia, […]
> 2. If the caller is an external language, […] In this case, we can enable a network protocol to transmit function pointers over the network, allowing both two side to access each other’s C API (this approach is inspired by MATLAB’s interoperability).
> 
> **Motivation**  
> The need for TyJuliaCAPI arises from a technical challenge: **How can Julia interact with other external languages within the same process**?
> 
> We have [several] projects implemented in C++/C#, and calling Julia from native code is a crucial requirement.
> 
> So, how to call Julia from C#: The only solution to this problem is to use the C API.
> 
> However, although Julia provides a C API ([julia.h](https://github.com/JuliaLang/julia/blob/master/src/julia.h)), these APIs have the following issues: […]
> 
> 1. The API is unstable and changes with every version. […]
> 
> And we need a stable and generic Julia C API.
> 
> In the past, we have learned a useful technique from [PythonCall.jl](https://github.com/JuliaPy/PythonCall.jl) (referred to as GC Pooling), which PythonCall itself used to provide a set of interoperation mechanisms, surpassing similar projects in terms of stability and performance. Following this technique, we have implemented a stable and well-designed C API for Julia, which we call TyJuliaCAPI.
> 
> In TyJuliaCAPI, the lifecycle management is adopted in a manner similar to the [Python Stable C API](https://docs.python.org/3/c-api/stable.html) (the most widely applied and highly regarded C API design)  
> […]  
> The complete API list can be found in [`include/tyjuliacapi.hpp`](https://github.com/Suzhou-Tongyuan/TyJuliaCAPI.jl/blob/main/include/tyjuliacapi-docs.hpp).
> 
> **External Language Binding**
> 
> Currently, we have implemented a C# language binding generator, which can be found in `bindings/csharp.jl`