Does anyone knows how to setup a Julia package that uses Python via another imported package that includes the PyCall setup? Here a more detailed explanation:
I am developing a series of packages that depend of each other. The base library (Sleipnir.jl) serves as the package that contains basic structures and have the configuration using PyCall to set up the Python environment. On top of this library, there is Huginn.jl which implements certain features using elements from the base library Sleipnir. As expected, this second library imports the first one.
All the Python setup has been done directly in the base library Sleipnir, by including the following statement in the main Julia script Sleipnir.jl
__precompile__() # this module is safe to precompile module Sleipnir ... import Pkg using PyCall ... const netCDF4::PyObject = PyNULL() const cfg::PyObject = PyNULL() const utils::PyObject = PyNULL() const workflow::PyObject = PyNULL() const tasks::PyObject = PyNULL() const global_tasks::PyObject = PyNULL() const graphics::PyObject = PyNULL() const bedtopo::PyObject = PyNULL() const millan22::PyObject = PyNULL() const MBsandbox::PyObject = PyNULL() const salem::PyObject = PyNULL() # Essential Python libraries const xr::PyObject = PyNULL() const rioxarray::PyObject = PyNULL() const pd::PyObject = PyNULL() ... include("setup/config.jl") ... end # module
and the following configuration in
function __init__() # Create structural folders if needed OGGM_path = joinpath(homedir(), "Python/OGGM_data") if !isdir(OGGM_path) mkpath(OGGM_path) end println("Initializing Python libraries...") # Load Python packages try copy!(netCDF4, pyimport("netCDF4")) copy!(cfg, pyimport("oggm.cfg")) copy!(utils, pyimport("oggm.utils")) copy!(workflow, pyimport("oggm.workflow")) copy!(tasks, pyimport("oggm.tasks")) copy!(global_tasks, pyimport("oggm.global_tasks")) copy!(graphics, pyimport("oggm.graphics")) copy!(bedtopo, pyimport("oggm.shop.bedtopo")) copy!(millan22, pyimport("oggm.shop.millan22")) copy!(MBsandbox, pyimport("MBsandbox.mbmod_daily_oneflowline")) copy!(salem, pyimport("salem")) copy!(pd, pyimport("pandas")) copy!(xr, pyimport("xarray")) copy!(rioxarray, pyimport("rioxarray")) catch @warn "It looks like you have not installed and/or activated the virtual Python environment. \n Please follow the guidelines in: https://github.com/ODINN-SciML/ODINN.jl#readme" end end function clean() atexit() do run(`$(Base.julia_cmd())`) end exit() end
All this setup works perfectly well for this library and the tests run perfectly well using CI and making the setup of the conda environment (see .github/workflows/CI.yml).
Now, the second library Huginn.jl is build on top of the first one and just makes an import
@reexport using Sleipnir
without doing any further setup of the Python dependencies.
What is the problem? When I ran the tests of this second library in local, I can perfectly well run the test using the Python configuration that is coming from the first library, Sleipnir. Now, when I add this to the package repository and I ran the test using CI (same setup), the test not pass because the Python setup is not done properly, even when the conda environments has been properly configured. You can see the stacktrace of the error here.
Do you have any idea of what can be causing this? Does anyone have examples of multiple libraries communicating with each other and importing the PyCall config from a dependency? Any help here is useful! I am trying to make a release of this library but right now I am stuck with not knowing what fails.