CachePath.jl allows using PyCall with two libpythons and no recompilation

General Usage
1

This package (unregistered) CachePath.jl allows you to specify a depot for saving and loading a single (or several) precompilation cache files (.ji files).

This branch of PyCall.jl allows you to save the metadata on which version of Python it is built with, eg which libpython, in a directory other than the source installation of PyCall.jl in the depot. That is “deps.jl” may be stored outside of ~/.julia/packages/PyCall.

Together, these allow you to use PyCall with two different libpythons without any recompilation:

shell> julia-latest -q
julia> using CachePath;
[ Info: Precompiling CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f]
[ Info: Skipping precompilation since __precompile__(false). Importing CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f].

julia> const PyCall = Base.require("PyCall", "../adepot")
PyCall

julia> PyCall.@pyimport sys

julia> sys.executable
"/home/lapeyre/.julia/conda/3/bin/python"

julia> PyCall.@pyimport numpy

julia> numpy.random.random(2)
2-element Vector{Float64}:
 0.1870384197176972
 0.3809703924611425

julia>

shell> julia-latest -q
julia> using CachePath;
[ Info: Precompiling CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f]
[ Info: Skipping precompilation since __precompile__(false). Importing CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f].

julia> const PyCall = Base.require("PyCall", "../adepot2")
PyCall

julia> PyCall.@pyimport sys

julia> sys.executable
"/usr/sbin/python"

julia> PyCall.@pyimport numpy

julia> numpy.random.random(2)
2-element Vector{Float64}:
 0.8474701385653681
 0.6679391959655362

I just threw this together, so it’s not super-convenient or bulletproof, but it does work.

EDIT: This has been fixed: In particular CachePath.jl evaluates everything in Base, and so is not precompiled, just becase copying code out of Base into another module can be more difficult than you might think. But, it’s possible.

EDIT: (see comments below in this thread)
Note that using the alternative PythonCall.jl and its compantion juliacall with different libpythons (and python exectuables) does not trigger recompilation of the cache file for PythonCall.jl.
Anticipating the question: PythonCall.jl is more transparent and automatic about recompilation upon switching libpython versions, but it still does recompilation every time.

Changing deps.jl to conflict with the compiled cache file will cause recompilation upon requireing:

shell> julia-latest -q

# depot "adepot2" cached PyCall uses /usr/sbin/python
julia> ENV["PYCALL_DEPOT"] = "/home/lapeyre/.julia/dev/adepot2";

# Use the conda python installation instead
julia> ENV["PYTHON"] = "";

# writes deps.jl to specify conda python
julia> Pkg.build("PyCall")

julia> using CachePath;
[ Info: Precompiling CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f]
[ Info: Skipping precompilation since __precompile__(false). Importing CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f].

# PyCall is recompiled
julia> const PyCall = Base.require("PyCall", "../adepot2") # This will trigger recompilation
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
PyCall

julia> 

# If we rebuild with the same libpython and the same depot path `adepo2`, then no
# precompilation happens

shell>  julia-latest -q
julia> ENV["PYCALL_DEPOT"] = "/home/lapeyre/.julia/dev/adepot2";

julia> ENV["PYTHON"] = "";

julia> Pkg.build("PyCall")
    Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
    Building PyCall → `~/.julia/dev/PyCall/deps/build.log`

julia> using CachePath;
[ Info: Precompiling CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f]
[ Info: Skipping precompilation since __precompile__(false). Importing CachePath [99f2aab7-dba8-43e9-8e3a-7970522a3b8f].

julia> const PyCall = Base.require("PyCall", "../adepot2")
PyCall
1 Like
2

This is incorrect, PythonCall supports using different libpythons without any recompilation whatsoever. The libpython selection happens entirely at run time. I’d appreciate if you could correct this statement.

3

Thanks for the correction. I just checked using both PythonCall.jl from julia and juliacall from python. I was confused because the modification timestamp of the cache files are updated when they are loaded.

Unless I’m missing something, the documentation for how to use different libpythons with PythonCall could be improved. ​I find only one line on this in the source file python.md:

To force PythonCall to use the same Python interpreter as PyCall, set the environment variable JULIA_PYTHONCALL_EXE to "@PyCall".

I don’t see JULIA_PYTHONCALL_EXE mentioned anywhere else. In fact, you can set JULIA_PYTHONCALL_EXE to the path of a python executable.

I just now found how to choose a libpython with PythonCall, so I don’t have much experience. But, I think not having to worry about recompilation and restarting is a huge advantage of PythonCall over PyCall. This is especially important for a Python user who is new to Julia and wants to stay in Python to the extent possible. Getting the minimum understanding of Pkg.build, libpython, and compilation caches required to use pyjulia and PyCall.jl can be difficult even for regular Julia users.

You might imagine that knowing more about the libpython at compile time would allow faster calls between python and julia. But, in fact, the last time I checked, the overhead for calling julia from python was smaller with juliacall (Pythoncall.jl) than with julia (PyCall.jl) by a factor of two.