Python setup across multiple dependent packages using PyCall

Tooling Package Management
1

Hi,

Does anyone knows how to setup a Julia package that uses Python via another imported package that includes the PyCall setup? Here a more detailed explanation:

I am developing a series of packages that depend of each other. The base library (Sleipnir.jl) serves as the package that contains basic structures and have the configuration using PyCall to set up the Python environment. On top of this library, there is Huginn.jl which implements certain features using elements from the base library Sleipnir. As expected, this second library imports the first one.

All the Python setup has been done directly in the base library Sleipnir, by including the following statement in the main Julia script Sleipnir.jl

__precompile__() # this module is safe to precompile
module Sleipnir
...
import Pkg
using PyCall
...
const netCDF4::PyObject = PyNULL()
const cfg::PyObject = PyNULL()
const utils::PyObject = PyNULL()
const workflow::PyObject = PyNULL()
const tasks::PyObject = PyNULL()
const global_tasks::PyObject = PyNULL()
const graphics::PyObject = PyNULL()
const bedtopo::PyObject = PyNULL()
const millan22::PyObject = PyNULL()
const MBsandbox::PyObject = PyNULL()
const salem::PyObject = PyNULL()

# Essential Python libraries
const xr::PyObject = PyNULL()
const rioxarray::PyObject = PyNULL()
const pd::PyObject = PyNULL()
...
include("setup/config.jl")
...
end # module

and the following configuration in config.jl

function __init__()

    # Create structural folders if needed
    OGGM_path = joinpath(homedir(), "Python/OGGM_data")
    if !isdir(OGGM_path)
        mkpath(OGGM_path)
    end

    println("Initializing Python libraries...")

    # Load Python packages
    try
        copy!(netCDF4, pyimport("netCDF4"))
        copy!(cfg, pyimport("oggm.cfg"))
        copy!(utils, pyimport("oggm.utils"))
        copy!(workflow, pyimport("oggm.workflow"))
        copy!(tasks, pyimport("oggm.tasks"))
        copy!(global_tasks, pyimport("oggm.global_tasks"))
        copy!(graphics, pyimport("oggm.graphics"))
        copy!(bedtopo, pyimport("oggm.shop.bedtopo"))
        copy!(millan22, pyimport("oggm.shop.millan22"))
        copy!(MBsandbox, pyimport("MBsandbox.mbmod_daily_oneflowline"))
        copy!(salem, pyimport("salem"))
        copy!(pd, pyimport("pandas"))
        copy!(xr, pyimport("xarray"))
        copy!(rioxarray, pyimport("rioxarray"))
    catch
        @warn "It looks like you have not installed and/or activated the virtual Python environment. \n 
        Please follow the guidelines in: https://github.com/ODINN-SciML/ODINN.jl#readme"
    end

end

function clean()
    atexit() do
        run(`$(Base.julia_cmd())`)
    end
    exit()
 end

All this setup works perfectly well for this library and the tests run perfectly well using CI and making the setup of the conda environment (see .github/workflows/CI.yml).

Now, the second library Huginn.jl is build on top of the first one and just makes an import

@reexport using Sleipnir

without doing any further setup of the Python dependencies.

What is the problem? When I ran the tests of this second library in local, I can perfectly well run the test using the Python configuration that is coming from the first library, Sleipnir. Now, when I add this to the package repository and I ran the test using CI (same setup), the test not pass because the Python setup is not done properly, even when the conda environments has been properly configured. You can see the stacktrace of the error here.

Do you have any idea of what can be causing this? Does anyone have examples of multiple libraries communicating with each other and importing the PyCall config from a dependency? Any help here is useful! I am trying to make a release of this library but right now I am stuck with not knowing what fails.

2

In case someone finds this same issue, here is how it was finally solved thanks to the help of @mkitti !

The problem seems to be coming from the SSL certificates that Julia and Python were using. At the moment of running the test of Huginn.jl the test was failing because an incompatibility of the SSL certificates that Python was using with the conda setup and the one Julia was executing with the tests. This can be solved by forcing Python to use the same certificates that Julia is using. The Julia certificates can be obtained by simply doing

using NetworkOptions
println(NetworkOptions.ca_roots())

and the when running the Julia action just do

      - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : /home/runner/micromamba/envs/oggm_env/bin/python
          # The SSL certificate path can be readed from the action "Check Julia SSL certifications"
          SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt

The full workflow can be found in this CI workflow file.

2 Likes