PyCall and Conda inside GitHub Actions

Tooling
, ,
1

Hi all, I am having some trouble trying to run unit tests with a nice Julia code I wrote that has to call a python package for a couple functions.

My understanding was that as long as the environment used by Conda.jl and PyCall.jl was the same, I can add packages using Conda.jl no problem and then import them. My workflow is to have an init() function at the beginning of my module as below.

using PyCall
import Conda

function __init__()
    if !haskey(Conda._installed_packages_dict(),"crowdsourcephoto")
        Conda.add("crowdsourcephoto",channel="conda-forge")
    end
    py"""
    import crowdsource # the name of the package installed via the crowdsourcephoto feedstock
    """
end

When this is run before a github action to do the unit tests, it fails saying that there is no module crowdsourceā€¦ as if I installed the package to the wrong environment. But I checked that the Conda packages were being installed to ā€œ/home/runner/.julia/conda/3ā€, which I think is the right place given that I use the default PyCall at /home/runner/.julia/packages/PyCall/BD546/src/. Error code below.


ERROR: LoadError: InitError: PyError ($(Expr(:escape, :(ccall(#= /home/runner/.julia/packages/PyCall/BD546/src/pyeval.jl:38 =# @pysym(:PyEval_EvalCode), PyPtr, (PyPtr, PyPtr, PyPtr), o, globals, locals))))) <class 'ModuleNotFoundError'>

[448](https://github.com/andrew-saydjari/disCovErr.jl/runs/3959676507?check_suite_focus=true#step:5:448)ModuleNotFoundError("No module named 'crowdsource'")

On my local machine, I set the python environment to the defaults, add, rebuild, and all goes well. Butā€¦ I donā€™t expect to have to rebuild with just a Conda.add() so I thought this should work in the init() function as I have it now. Am I missing something?

2

Did you configure PyCall.jl to use Conda, e.g. by setting the environemnt variable PYHTON=""? Thatā€™s something I needed to do for some of my packages, e.g. https://github.com/ranocha/BSeries.jl/blob/d0db96175376c20b31d982565501ef3b592b7ebf/.github/workflows/CI.yml#L31

2 Likes
3

I had not. That saved me a lot of (additional) time. Thanks!

4

As an alternative, you may like to try out GitHub - cjdoris/PythonCall.jl: Python and Julia in harmony.. It handles Python versions and package dependencies for you so you donā€™t have to worry about this sort of thing. Try the master branch (I havenā€™t made a release for a while).

5

Hi @andrew-saydjari, As I understand it, you may be interested to do a check if PyCall is using Conda.jl by executing PyCall.conda in Julia Repl. If true - its Conda.jl, if false its ā€¦ not.

Hi @cjdoris, I am interested in your package. Your package looks ultra interesting. Thank you. Iā€™d like to ask a question wrt its usage as I am receiving ERROR: Python: ModuleNotFoundError.

I created a new Julia environment pythoncall_test and activated it. I added PythonCall. It created a new directory in my Julia environment called conda_env and installed conda 4.9.2 and Python 3.10.0 and created and activated a new conda environment */home/username/data/pythoncall_test/conda_env. I entered Julia shell and installed a packaged called python-cdo which is a python wrapper of Fortran software called Climate Data Operators (please find some details below) by conda install -c conda-forge python-cdo. The installation went correctly and I can see cdo and python-cdo present in packages in /home/username/data/pythoncall_test/conda_env/bin.

I used to use PyCall in the following way (this is from another machine so paths are different):

julia> using PyCall
julia> pycdo = pyimport("cdo")
PyObject <module 'cdo' from '/opt/julia/conda/3/lib/python3.9/site-packages/cdo.py'>
julia> cdo = pycdo.Cdo()
PyObject <cdo.Cdo object at 0x7f889b3939a0>

And I was able to interact with cdo software in Julia through its python-cdo wrapper.

And now, when I try to replicate the procedure in PythonCall I see the following ERROR:

julia> using PythonCall
julia> pycdo = pyimport("cdo")
ERROR: Python: ModuleNotFoundError: No module named 'cdo'
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/88QSa/src/err.jl:92
 [2] errcheck
   @ ~/.julia/packages/PythonCall/88QSa/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/88QSa/src/concrete/import.jl:11
 [4] top-level scope
   @ REPL[18]:1

Would it be possible to receive some advice? Are there any differences with regard to this part?

# Details of Climate Data Operators package
shell> cdo --version
Climate Data Operators version 1.9.10 (https://mpimet.mpg.de/cdo)
System: x86_64-conda-linux-gnu
CXX Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-c++ -fPIC -DPIC -g -O2 -std=c++11 -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix -fopenmp -pthread
CXX version : unknown
C Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-cc -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix -fopenmp -pthread -pthread
C version : unknown
F77 Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-gfortran -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix
F77 version : GNU Fortran (GCC) 9.4.0
Features: 376GB 64threads C++11 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ XML2 CURL FFTW3 SSE3
Libraries: HDF5/1.12.1 proj/8.1.1 xml2/2.9.12 curl/7.79.1
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 1.9.10
 cgribex library version : 1.9.5
 ecCodes library version : 2.23.0
  NetCDF library version : 4.8.1 of Aug 31 2021 00:05:39 $
    hdf5 library version : 1.12.1 threadsafe
    exse library version : 1.4.2
    FILE library version : 1.9.1
6

The docs could probably be improved in this regard, but donā€™t add packages by calling conda yourself. Either create a PythonCallDeps.toml file or use PythonCall.Deps.add.

See here: Guide Ā· PythonCall & JuliaCall

PythonCall assumes all dependencies are declared this way and will sometimes recreate the conda environment from scratch, so any manually installed packages will disappear.

7

Thank you for the reply.

The docs could probably be improved in this regard [ā€¦]

Nothing better than consult the creator directly. :slight_smile: With all due respect to PyCall, I do really like PythonCall.

I hope @andrew-saydjari does not mind I will try to ask some additional questions here. Pls be informed that should there be a need/will, I am ready to start a new topic.

I am on Julia Version 1.8.0-DEV.772 and PythonCall v0.4.0 GitHub - JuliaPy/PythonCall.jl: Python and Julia in harmony.

[ā€¦] donā€™t add packages by calling conda yourself. Either create a PythonCallDeps.toml file or use PythonCall.Deps.add.

So I understand that in case of PythonCallDeps.toml I will have two of those files:

[ Info: Found PythonCall dependencies at '/home/username/data/pythoncall_test1/PythonCallDeps.toml'
[ Info: Found PythonCall dependencies at '/home/username/.julia/packages/PythonCall/KiVdf/PythonCallDeps.toml'

The first one I can create myself or will be created automaticly when I use / add some dependencies i.e. by PythonCall.Deps.add(conda_packages= ["python-cdo=1.5.5"]) # the syntax is slightly different than PyCall as it requires [].

I did so like a line above and I got PythonCall.Deps.status() printing:

Status `~/data/pythoncall_test1/Project.toml`
  [5fb14364] OhMyREPL v0.5.10
  [6099a3de] PythonCall v0.4.0 `https://github.com/cjdoris/PythonCall.jl.git#master`

Status /home/username/data/pythoncall_test1/PythonCallDeps.toml
Conda packages:
  python-cdo =1.5.5

However, I am still getting ERROR: LoadError: Python: ModuleNotFoundError: No module named 'cdo' when trying pycdo = pyimport("cdo").

I am recalling that when using PyCall even thought executing julia> Conda.add("python-cdo", channel="conda-forge") I had to add this package again by shell> conda install -c conda-forge python-cdo otherwise it was similarly not working. I do not now why was that. Other thing I noticed, I am not sure if it makes any difference, is that when PythonCall is adding the package, in contrary to pip and python I see lack of '' like in the line below.

[ Info: Running conda install --yes --no-channel-priority --freeze-installed --channel defaults ā€˜pip>=18ā€™ python-cdo=1.5.5 ā€˜python>=3.5,<4ā€™ in /home/username/data/pythoncall_test1/conda_env environment

The other thing, not related to python-cdo is following info when trying to recreate the environment by PythonCall.Deps.add(create=true). This I guess might be related to my global conda, however, I am not sure.

[ Info: Found PythonCall dependencies at '/home/username/data/pythoncall_test/PythonCallDeps.toml'
[ Info: Found PythonCall dependencies at '/home/username/.julia/packages/PythonCall/KiVdf/PythonCallDeps.toml'
[ Info: Running `conda env remove --yes --prefix /home/username/data/pythoncall_test/conda_env` in root environment

Remove all packages in environment /home/username/data/pythoncall_test/conda_env:

WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551500000001a.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5517e0000001c.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551580000001b.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551880000001e.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5518b0000001d.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5517b00000019.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5516600000018.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/.nfs0000000094f7e6640000001f.  Please remove this file manually (you may need to reboot to free file handles)
[ Info: Running `conda create --yes --no-default-packages --no-channel-priority --prefix /home/username/data/pythoncall_test/conda_env --channel defaults 'pip>=18' python-cdo=1.5.5 'python>=3.5,<4'` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/username/data/pythoncall_test/conda_env

  added / updated specs:
    - pip[version='>=18']
    - python-cdo=1.5.5
    - python[version='>=3.5,<4']

Is there any chance for some additional advice, pls?

8

Hm that is indeed odd. Can you start a new topic for this please? Or make a GitHub issue.

9

Hi, I did some additional testing. I found no problems this time. First on a new Linux system with full rights, Julia 1.6 and slightly more up to date version of PythonCall than before (v0.4.1 vs v0.4.0) and everything worked fine there. Then on the same system where I did encounter some problems. This time I payed more attention to the correct setup of Julia, global Conda and VS Code environments. I used PythonCall v0.4.2 this time. I tested on Julia 1.7beta3 and 1.8.0-DEV.772 as well as on 1.6 installed in a separate Conda environment. And also this time I did not encounter any problems. I am not sure for 100%, however, as for now, I would say that it was more related to proper setup of Julia, global Conda and VS Code environments than PythonCall versions. Also its important to restart Julia after adding new packages, otherwise it does not work (I am pretty sure I did it previously as well at least on most of the occasions). I will continue using PythonCall as I like the concept a lot. Its solving most of my problems related to reproduction of the setup on various remote machines including some with global non-writable Conda depot and easily opening the whole Python and Conda ecosystem. Should I encounter any serious problems or any WARNINGS like conda.gateways.disk.delete:unlink_or_rename_to_trash(143) then I will open a new topic here or at GitHub and ask for the advice. Thanks so much once again!

10

I looked into the issue you had, it turns out a Conda flag I was using does not behave the way I expected, so I removed it in v0.4.2. Iā€™m glad itā€™s working now and that you find it useful!

11

Thanks a lot once again. Great package!