# PyCall and Conda inside GitHub Actions

**URL:** <https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128>\
**Category:** Tooling\
**Tags:** pycall, github-actions, conda\
**Created:** [October 21, 2021, 5:32am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128 "2021-10-21T05:32:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrew-saydjari](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrew-saydjari/32/19896_2.png) [@andrew-saydjari](https://discourse.julialang.org/u/andrew-saydjari)\
**Post date:** [October 21, 2021, 5:32am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/1 "2021-10-21T05:32:39Z")

</div>

Hi all, I am having some trouble trying to run unit tests with a nice Julia code I wrote that has to call a python package for a couple functions.

My understanding was that as long as the environment used by Conda.jl and PyCall.jl was the same, I can add packages using Conda.jl no problem and then import them. My workflow is to have an `init()` function at the beginning of my module as below.

```julia
using PyCall
import Conda

function __init__ ()
    if !haskey(Conda._installed_packages_dict(),"crowdsourcephoto")
        Conda.add("crowdsourcephoto",channel="conda-forge")
    end
    py"""
    import crowdsource # the name of the package installed via the crowdsourcephoto feedstock
    """
end

```

When this is run before a github action to do the unit tests, it fails saying that there is no module `crowdsource`… as if I installed the package to the wrong environment. But I checked that the Conda packages were being installed to “/home/runner/.julia/conda/3”, which I think is the right place given that I use the default PyCall at /home/runner/.julia/packages/PyCall/BD546/src/. Error code below.

```julia

ERROR: LoadError: InitError: PyError ($(Expr(:escape, :(ccall(#= /home/runner/.julia/packages/PyCall/BD546/src/pyeval.jl:38 =# @pysym(:PyEval_EvalCode), PyPtr, (PyPtr, PyPtr, PyPtr), o, globals, locals))))) <class 'ModuleNotFoundError'>

[448](https://github.com/andrew-saydjari/disCovErr.jl/runs/3959676507?check_suite_focus=true#step:5:448)ModuleNotFoundError("No module named 'crowdsource'")

```

On my local machine, I set the python environment to the defaults, add, rebuild, and all goes well. But… I don’t expect to have to rebuild with just a Conda.add() so I thought this should work in the init() function as I have it now. Am I missing something?

---

<div class="post-metadata">

**Author:** ![ranocha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ranocha/32/35588_2.png) [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Post date:** [October 21, 2021, 5:45am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/2 "2021-10-21T05:45:42Z")

</div>

Did you configure PyCall.jl to use Conda, e.g. by setting the environemnt variable `PYHTON=""`? That’s something I needed to do for some of my packages, e.g. [https://github.com/ranocha/BSeries.jl/blob/d0db96175376c20b31d982565501ef3b592b7ebf/.github/workflows/CI.yml#L31](https://github.com/ranocha/BSeries.jl/blob/d0db96175376c20b31d982565501ef3b592b7ebf/.github/workflows/CI.yml#L31)

---

<div class="post-metadata">

**Author:** ![andrew-saydjari](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrew-saydjari/32/19896_2.png) [@andrew-saydjari](https://discourse.julialang.org/u/andrew-saydjari)\
**Post date:** [October 21, 2021, 6:29am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/3 "2021-10-21T06:29:23Z")

</div>

I had not. That saved me a lot of (additional) time. Thanks!

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 21, 2021, 6:32am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/4 "2021-10-21T06:32:08Z")

</div>

As an alternative, you may like to try out [GitHub - cjdoris/PythonCall.jl: Python and Julia in harmony.](https://github.com/cjdoris/PythonCall.jl). It handles Python versions and package dependencies for you so you don’t have to worry about this sort of thing. Try the master branch (I haven’t made a release for a while).

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [October 21, 2021, 4:46pm UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/5 "2021-10-21T16:46:05Z")

</div>

Hi @andrew-saydjari, As I understand it, you may be interested to do a check if PyCall is using Conda.jl by executing PyCall.conda in Julia Repl. If true - its Conda.jl, if false its … not.

Hi @cjdoris, I am interested in your package. Your package looks ultra interesting. Thank you. I’d like to ask a question wrt its usage as I am receiving `ERROR: Python: ModuleNotFoundError`.

I created a new Julia environment `pythoncall_test` and activated it. I added PythonCall. It created a new directory in my Julia environment called `conda_env` and installed conda 4.9.2 and Python 3.10.0 and created and activated a new conda environment `*/home/username/data/pythoncall_test/conda_env`. I entered Julia shell and installed a packaged called python-cdo which is a python wrapper of Fortran software called Climate Data Operators (please find some details below) by `conda install -c conda-forge python-cdo`. The installation went correctly and I can see `cdo` and `python-cdo` present in packages in `/home/username/data/pythoncall_test/conda_env/bin`.

I used to use PyCall in the following way (this is from another machine so paths are different):

```julia
julia> using PyCall
julia> pycdo = pyimport("cdo")
PyObject <module 'cdo' from '/opt/julia/conda/3/lib/python3.9/site-packages/cdo.py'>
julia> cdo = pycdo.Cdo()
PyObject <cdo.Cdo object at 0x7f889b3939a0>

```

And I was able to interact with cdo software in Julia through its python-cdo wrapper.

And now, when I try to replicate the procedure in PythonCall I see the following ERROR:

```julia
julia> using PythonCall
julia> pycdo = pyimport("cdo")
ERROR: Python: ModuleNotFoundError: No module named 'cdo'
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/88QSa/src/err.jl:92
 [2] errcheck
   @ ~/.julia/packages/PythonCall/88QSa/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/88QSa/src/concrete/import.jl:11
 [4] top-level scope
   @ REPL[18]:1

```

Would it be possible to receive some advice? Are there any differences with regard to this part?

```julia
# Details of Climate Data Operators package
shell> cdo --version
Climate Data Operators version 1.9.10 (https://mpimet.mpg.de/cdo)
System: x86_64-conda-linux-gnu
CXX Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-c++ -fPIC -DPIC -g -O2 -std=c++11 -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix -fopenmp -pthread
CXX version : unknown
C Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-cc -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix -fopenmp -pthread -pthread
C version : unknown
F77 Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1633096547727/_build_env/bin/x86_64-conda-linux-gnu-gfortran -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/data/pythoncall_test/conda_env/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cdo_1633096547727/work=/usr/local/src/conda/cdo-1.9.10 -fdebug-prefix-map=/home/username/data/pythoncall_test/conda_env=/usr/local/src/conda-prefix
F77 version : GNU Fortran (GCC) 9.4.0
Features: 376GB 64threads C++11 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ XML2 CURL FFTW3 SSE3
Libraries: HDF5/1.12.1 proj/8.1.1 xml2/2.9.12 curl/7.79.1
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 1.9.10
 cgribex library version : 1.9.5
 ecCodes library version : 2.23.0
  NetCDF library version : 4.8.1 of Aug 31 2021 00:05:39 $
    hdf5 library version : 1.12.1 threadsafe
    exse library version : 1.4.2
    FILE library version : 1.9.1

```

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 21, 2021, 7:42pm UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/6 "2021-10-21T19:42:49Z")

</div>

The docs could probably be improved in this regard, but don’t add packages by calling conda yourself. Either create a `PythonCallDeps.toml` file or use `PythonCall.Deps.add`.

See here: [Guide · PythonCall & JuliaCall](https://cjdoris.github.io/PythonCall.jl/dev/pythoncall/#Managing-Python-dependencies)

PythonCall assumes all dependencies are declared this way and will sometimes recreate the conda environment from scratch, so any manually installed packages will disappear.

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [October 21, 2021, 11:40pm UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/7 "2021-10-21T23:40:41Z")

</div>

Thank you for the reply.

> The docs could probably be improved in this regard […]

Nothing better than consult the creator directly. 🙂 With all due respect to PyCall, I do really like PythonCall.

I hope @andrew-saydjari does not mind I will try to ask some additional questions here. Pls be informed that should there be a need/will, I am ready to start a new topic.

I am on `Julia Version 1.8.0-DEV.772` and `PythonCall v0.4.0 `[GitHub - JuliaPy/PythonCall.jl: Python and Julia in harmony.](https://github.com/cjdoris/PythonCall.jl.git#master.%60)

> […] don’t add packages by calling conda yourself. Either create a PythonCallDeps.toml file or use PythonCall.Deps.add.

So I understand that in case of `PythonCallDeps.toml` I will have two of those files:

```julia
[ Info: Found PythonCall dependencies at '/home/username/data/pythoncall_test1/PythonCallDeps.toml'
[ Info: Found PythonCall dependencies at '/home/username/.julia/packages/PythonCall/KiVdf/PythonCallDeps.toml'

```

The first one I can create myself or will be created automaticly when I use / add some dependencies i.e. by `PythonCall.Deps.add(conda_packages= ["python-cdo=1.5.5"]) # the syntax is slightly different than PyCall as it requires []`.

I did so like a line above and I got `PythonCall.Deps.status()` printing:

```julia
Status `~/data/pythoncall_test1/Project.toml`
  [5fb14364] OhMyREPL v0.5.10
  [6099a3de] PythonCall v0.4.0 `https://github.com/cjdoris/PythonCall.jl.git#master`

Status /home/username/data/pythoncall_test1/PythonCallDeps.toml
Conda packages:
  python-cdo =1.5.5

```

However, I am still getting `ERROR: LoadError: Python: ModuleNotFoundError: No module named 'cdo'` when trying `pycdo = pyimport("cdo")`.

I am recalling that when using PyCall even thought executing `julia> Conda.add("python-cdo", channel="conda-forge")` I had to add this package again by` shell> conda install -c conda-forge python-cdo` otherwise it was similarly not working. I do not now why was that. Other thing I noticed, I am not sure if it makes any difference, is that when PythonCall is adding the package, in contrary to pip and python I see lack of `''` like in the line below.

`[ Info: Running `conda install --yes --no-channel-priority --freeze-installed --channel defaults ‘pip\>=18’ python-cdo=1.5.5 ‘python\>=3.5,\<4’` in /home/username/data/pythoncall_test1/conda_env environment`

The other thing, not related to python-cdo is following info when trying to recreate the environment by `PythonCall.Deps.add(create=true)`. This I guess might be related to my global conda, however, I am not sure.

```julia
[ Info: Found PythonCall dependencies at '/home/username/data/pythoncall_test/PythonCallDeps.toml'
[ Info: Found PythonCall dependencies at '/home/username/.julia/packages/PythonCall/KiVdf/PythonCallDeps.toml'
[ Info: Running `conda env remove --yes --prefix /home/username/data/pythoncall_test/conda_env` in root environment

Remove all packages in environment /home/username/data/pythoncall_test/conda_env:

WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551500000001a. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5517e0000001c. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551580000001b. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c551880000001e. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5518b0000001d. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5517b00000019. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/python3.10/lib-dynload/.nfs0000000092c5516600000018. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /home/username/data/pythoncall_test/conda_env/lib/.nfs0000000094f7e6640000001f. Please remove this file manually (you may need to reboot to free file handles)
[ Info: Running `conda create --yes --no-default-packages --no-channel-priority --prefix /home/username/data/pythoncall_test/conda_env --channel defaults 'pip>=18' python-cdo=1.5.5 'python>=3.5,<4'` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/username/data/pythoncall_test/conda_env

  added / updated specs:
    - pip[version='>=18']
    - python-cdo=1.5.5
    - python[version='>=3.5,<4']

```

Is there any chance for some additional advice, pls?

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 22, 2021, 7:38am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/8 "2021-10-22T07:38:57Z")

</div>

Hm that is indeed odd. Can you start a new topic for this please? Or make a GitHub issue.

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [October 24, 2021, 6:54pm UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/9 "2021-10-24T18:54:46Z")

</div>

Hi, I did some additional testing. I found no problems this time. First on a new Linux system with full rights, Julia 1.6 and slightly more up to date version of PythonCall than before (v0.4.1 vs v0.4.0) and everything worked fine there. Then on the same system where I did encounter some problems. This time I payed more attention to the correct setup of Julia, global Conda and VS Code environments. I used PythonCall v0.4.2 this time. I tested on Julia 1.7beta3 and 1.8.0-DEV.772 as well as on 1.6 installed in a separate Conda environment. And also this time I did not encounter any problems. I am not sure for 100%, however, as for now, I would say that it was more related to proper setup of Julia, global Conda and VS Code environments than PythonCall versions. Also its important to restart Julia after adding new packages, otherwise it does not work (I am pretty sure I did it previously as well at least on most of the occasions). I will continue using PythonCall as I like the concept a lot. Its solving most of my problems related to reproduction of the setup on various remote machines including some with global non-writable Conda depot and easily opening the whole Python and Conda ecosystem. Should I encounter any serious problems or any WARNINGS like conda.gateways.disk.delete:unlink\_or\_rename\_to\_trash(143) then I will open a new topic here or at GitHub and ask for the advice. Thanks so much once again!

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 24, 2021, 9:10pm UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/10 "2021-10-24T21:10:11Z")

</div>

I looked into the issue you had, it turns out a Conda flag I was using does not behave the way I expected, so I removed it in v0.4.2. I’m glad it’s working now and that you find it useful!

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [October 26, 2021, 1:27am UTC](https://discourse.julialang.org/t/pycall-and-conda-inside-github-actions/70128/11 "2021-10-26T01:27:05Z")

</div>

Thanks a lot once again. Great package!