Any working way to call Python from Julia compiled app?

General Usage
, , , ,
1

I tried to compile a relocatable app with PackageCompiler.create_app and call Python from it.

  1. With PyCall.jl - there is a problem that compiled binary has an absolute path to python library on dev machine (on my Windows machine it is C:\Users\gvg\.julia\conda\3\python39.dll), so when I’m on another machine with different user - it can’t find the python by this path. See issue here: https://github.com/JuliaPy/PyCall.jl/issues/981
  2. With PythonCall.jl - there is another error after compilation, looks like it can’t find Manifes.toml in compiled project files. See issue here: https://github.com/cjdoris/PythonCall.jl/issues/146

Any other ways I’ve missed?
With option 1, maybe I can set another python path before compilation, so it is not bound to a specific user?

2

I’m also finding PythonCall and PackageCompiler not working well together. It doesn’t use the JULIA_PYTHONCALL_EXE environment variable when compiled. For large sysimage with all packages included, it failed all together when starting julia with the compiled image:

julia>julia --sysimage=mysysimage.so
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=AssertionError(msg=“CTX.which == :PyCall”))
#34 at ….julia\packages\PythonCall\XgP8G\src\cpython\context.jl:135
with_gil at ….julia\packages\PythonCall\XgP8G\src\cpython\gil.jl:10 [inlined]
with_gil at ….julia\packages\PythonCall\XgP8G\src\cpython\gil.jl:9 [inlined]
init_context at ….julia\packages\PythonCall\XgP8G\src\cpython\context.jl:132
init at ….julia\packages\PythonCall\XgP8G\src\cpython\CPython.jl:21
jfptr___init___70514 at …\julia\mysysimage.so (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:73
_finish_julia_init at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:796
jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:695
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

3

I’ve rencently got PyCall.jl portable by replacing the dll finding process with reading environment variables.

Conda.jl is harder to get work. I left it remain the wrong configs (local configs), but replacing all related variables with reading references which will be setup when __init__ PyCall module.

1 Like
4

PythonCall now supports this.

I made this small package for configuration.

Things will work smoothly after this PR gets merges

2 Likes
5

I made a similar PR to PyCall.jl. It relies on another package as well, CachePath.jl. It would be great to put together a doc on the pros and cons of various approaches, including startup times, etc. Many of the important questions are not documented (or not clearly), so it will take a bit of work.

1 Like
6

I’ve tried to add JuliaPythonAdaptor into this small package to test compilation, but it failed with the same error as in Readme.md:

7

The package has now run into some issue and fails at configuring when loaded as a precompiled package. If precompiled, JuliaPythonAdaptor is always loaded after PythonCall…

8

Install JuliaPythonAdaptor via pip and try

python -c "import JuliaPythonAdaptor;import os;del os.environ['JULIA_PYTHONCALL_LIBPTR'];os.system('path/to/you_app_exe')"
9

On the python side?
Path to which executable should I provide at os.system('path/to/you_exe'), python or julia?

10

The path to the executable you built with PackageCompiler.

11

Seems like python doesn’t recognize package name:

PS I:\> python -c "import JuliaPythonAdaptor;import os;del os.environ['JULIA_PYTHONCALL_LIBPTR'];os.system('C:\\_KTAuto\\backend\\conda\\3\\python.exe')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'JuliaPythonAdaptor'
12

Ok, now I try to install it with pip and get the following:

PS I:\> python -m pip install https://github.com/thautwarm/JuliaPythonAdaptor.jl
Collecting https://github.com/thautwarm/JuliaPythonAdaptor.jl
  Downloading https://github.com/thautwarm/JuliaPythonAdaptor.jl
     \ 183.2 kB 381.8 kB/s 0:00:00
  ERROR: Cannot unpack file C:\Users\gvg\AppData\Local\Temp\pip-unpack-u150j3_z\JuliaPythonAdaptor.jl (downloaded from C:\Users\gvg\AppData\Local\Temp\pip-req-build-wl7iqp0o, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of C:\Users\gvg\AppData\Local\Temp\pip-req-build-wl7iqp0o
13

Have you tried pip install JuliaPythonAdaptor?

14

Ok, I have installed it on python side and re-compiled app once more, but it keeps running with the same error. Should I change something in this particular script?

python -c "import JuliaPythonAdaptor;import os;del os.environ['JULIA_PYTHONCALL_LIBPTR'];os.system('C:\\_KTAuto\\backend\\conda\\3\\python.exe')"

julia --project=@. --startup-file=no -e '
ENV["JP_ADAPTOR_PY_EXE"] = "C:\\_KTAuto\\backend\\conda\\3\\python.exe";
using PackageCompiler;
PackageCompiler.create_app(pwd(), "PythonCallCompiled";
    cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)",
    include_transitive_dependencies=false,
    filter_stdlibs=true,
    precompile_execution_file=["test/runtests.jl"])
'

Or maybe there are some variables that I can check after importing Julia package?