I initially ran into this error trying to use ClimateTools, but digging down into it, I think it’s a problem with how conda is setup within my Julia installation and outside of it. I have a local install of miniconda with a couple of environments setup, but conda is setup to not be activated by default. I’m trying to have Julia use Conda.jl to setup its own clean build of any necessary python packages instead of pointing Julia to my miniconda envs. If I try and build PyCall I’m getting this error:
julia> Pkg.build("PyCall")
Building Conda ─→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`
┌ Error: Error building `PyCall`:
│ Traceback (most recent call last):
│ File "/home/gregorylemieux/.julia/conda/3/bin/conda", line 12, in <module>
│ from conda.cli import main
│ ModuleNotFoundError: No module named 'conda'
│ ┌ Info: Using the Python distribution in the Conda package by default.
│ └ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
│ [ Info: Running `conda install -y numpy` in root environment
│ ERROR: LoadError: failed process: Process(setenv(`/home/gregorylemieux/.julia/conda/3/bin/conda install -y numpy`,["_CE_M=", "PATH=/home/gregorylemieux/local/miniconda3/condabin:/home/gregorylemieux/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/cuda/bin:/home/gregorylemieux/texlive/2018/bin/x86_64-linux:/home/gregorylemieux/pandoc/bin:/home/gregorylemieux/local/hdf5-1.10.5/bin:/opt/gurobi810/linux64/bin:/home/gregorylemieux/local/netcdf-c-4.6.3/bin:/home/gregorylemieux/local/netcdf-fortran-4.4.5/bin", "DEFAULTS_PATH=/usr/share/gconf/pop.default.path", "QT_ACCESSIBILITY=1", "PAPERSIZE=letter", "LD_LIBRARY_PATH=:/usr/local/lib/:/home/gregorylemieux/local/hdf5-1.10.5/lib:/opt/gurobi810/linux64/lib:/home/gregorylemieux/local/netcdf-c-4.6.3/lib:/home/gregorylemieux/local/netcdf-fortran-4.4.5/lib", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus", "MANPATH=:/home/gregorylemieux/texlive/2018/texmf-dist/doc/man", "XDG_SESSION_DESKTOP=pop", "SSH_AGENT_PID=2821" … "LC_NAME=en_US.UTF-8", "NETCDF_C_HOME=/home/gregorylemieux/local/netcdf-c-4.6.3", "JOURNAL_STREAM=9:54434", "LC_IDENTIFICATION=en_US.UTF-8", "JULIA_LOAD_PATH=@:/tmp/jl_HBbIYJ", "QT4_IM_MODULE=ibus", "WINDOWPATH=2", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:", "OPENBLAS_MAIN_FREE=1", "NETCDF_FORTRAN_PATH=/home/gregorylemieux/local/netcdf-fortran-4.4.5"]), ProcessExited(1)) [1]
│
│ Stacktrace:
│ [1] pipeline_error at ./process.jl:525 [inlined]
│ [2] run(::Cmd; wait::Bool) at ./process.jl:440
│ [3] run at ./process.jl:438 [inlined]
│ [4] runconda(::Cmd, ::String) at /home/gregorylemieux/.julia/packages/Conda/3rPhK/src/Conda.jl:114
│ [5] add(::String, ::String; channel::String) at /home/gregorylemieux/.julia/packages/Conda/3rPhK/src/Conda.jl:188
│ [6] add at /home/gregorylemieux/.julia/packages/Conda/3rPhK/src/Conda.jl:187 [inlined] (repeats 2 times)
│ [7] top-level scope at /home/gregorylemieux/.julia/packages/PyCall/zqDXB/deps/build.jl:84
│ [8] include(::String) at ./client.jl:457
│ [9] top-level scope at none:5
│ in expression starting at /home/gregorylemieux/.julia/packages/PyCall/zqDXB/deps/build.jl:43
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:942
Here’s what I’ve got for the environment variables in Julia:
julia> ENV["PYTHON"]
""
julia> ENV["CONDA_PYTHON_EXE"]
"/home/gregorylemieux/local/miniconda3/bin/python"
julia> ENV["CONDA_EXE"]
"/home/gregorylemieux/local/miniconda3/bin/conda"
Note: CONDA_JL_HOME is not set.
I’m running Julia 1.5.0 on Pop_OS 20.04.
Update: I gave up trying to get Julia to use it’s own conda and decided to create a conda_jl environment with my existing miniconda install per the instructions in the Conda.jl README. PyCall still had issues trying to import the necessary modules and the error output noted that it was pointing to the root installation:
julia> using ClimateTools
[ Info: Precompiling ClimateTools [4f4ee721-4970-5af2-8560-6c1d960e3231]
ERROR: InitError: PyError (PyImport_ImportModule
The Python package scipy.interpolate could not be found by pyimport. Usually this means
that you did not install scipy.interpolate in the Python version being used by PyCall.
PyCall is currently configured to use the Python version at:
/usr/bin/python3
...
julia> PyCall.python
"/usr/bin/python3"
Somewhat at a loss here, I went back to reading the instructions regarding Specifying the Python version. I decided on a whim to set ENV["PYTHON"]="" (which wasn’t set at all previously) which the documentations says will:
force Julia to use its own Python distribution, via Conda
After rebuilding PyCall it appears to be using my pre-existing miniconda environment python install:
julia> PyCall.python
"/home/gregorylemieux/local/miniconda3/envs/conda_jl/bin/python"
julia> PyCall.conda
true
The thing that confused me here is that based on the language in the PyCall.jl README above, I assumed that this would not work as I wasn’t using Julia’s own python distribution, I was using the python install via the new environment in my pre-existing miniconda installation. But obviously, my assumption was wrong given that PyCall.conda is true.
I think the thing here that was tripping me up is understanding what is meant by Julia having it’s own conda installation in relation to the above quote about have PyCall use Julia’s python distribution via Conda.jl. I assumed that I could force Julia to install it’s own version of conda (separate from my pre-existing installation) and that Julia would install and use python from that conda install. Is this correct?
If you close all Julia sessions, and then open a brand-new Julia session, and run the following commends:
julia> ENV["PYTHON"] = ""
julia> import Pkg
julia> Pkg.build("PyCall")
Then Julia should install its own Python, not use any Python from your system…
If this is not the behavior that you are seeing, this is a bug. Can you open a bug report?
Oh, I see you’ve been setting the CONDA_JL_HOME environment variable.
PyCall.jl obeys what Conda.jl tells it. So if you tell Conda.jl to use your own Conda, then PyCall.jl will install Python into this Conda, even if you set PYTHON to "".
Initially I wasn’t using CONDA_JL_HOME. Towards the end of my first post:
Note: CONDA_JL_HOME is not set.
I only switched to trying that out of desperation since I couldn’t seem to get Julia to use it’s own python conda install. I’ll try and replicate the issue and post to a bug report.
Yeah I do think this is a bug in Conda.jl and/or PyCall.jl.
Make sure that you can reproduce this from scratch in a fresh environment. By which I mean:
- Back up your current
~/.julia folder, e.g. cp ~/.julia ~/.julia.backup
- Delete the
~/.julia folder, i.e. rm -rf ~/.julia.
- Make sure that in your shell, you don’t have any environment variables set that begin with
CONDA, just to be safe.
- Then start Julia and do:
julia> ENV["PYTHON"] = ""
julia> import InteractiveUtils
julia> InteractiveUtils.versioninfo(; verbose = true)
julia> import Pkg
julia> Pkg.add("PyCall")
julia> import PyCall
And then proceed with whatever code that you want to run.
If you can still reproduce this bug, please open a bug report on PyCall.jl.
Thanks for the help. I wasn’t able to replicate the issue that I was seeing following your instructions:
julia> PyCall.conda
true
julia> PyCall.python
"/home/gregorylemieux/.julia/conda/3/bin/python"
Note that I tested this with and without CONDA env variables set in the shell and it worked either way.
I got to wondering if the issue is dependent on the order of operations. If I add PyCall prior to setting ENV["PYTHON"] = "" then try and build PyCall after setting that environment variable I get:
julia> PyCall.conda
false
julia> PyCall.python
"python3"
1 Like
This is interesting.
So you are saying that this:
julia> import Pkg
julia> ENV["PYTHON"] = ""
julia> Pkg.add("PyCall")
julia> Pkg.build("PyCall")
julia> import PyCall
julia> PyCall.python
julia> PyCall.conda
Gives different results to this:
julia> import Pkg
julia> Pkg.add("PyCall")
julia> ENV["PYTHON"] = ""
julia> Pkg.build("PyCall")
julia> import PyCall
julia> PyCall.python
julia> PyCall.conda
Is that right?
I’m guessing you did import PyCall or using PyCall in between the add and build steps. Once the PyCall module is loaded, re-building won’t change the currently loaded module — you have to restart Julia to re-load PyCall from a new build.
1 Like
Ah, ok, I think I see where I’m messing up. If I follow your steps explicitly I get the same results, ie:
julia> PyCall.python
"/home/gregorylemieux/.julia/conda/3/bin/python"
julia> PyCall.conda
true
If I do the following I can replicate the issue:
gregorylemieux@pop-os:~$ rm -rf .julia
gregorylemieux@pop-os:~$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.1 (2020-08-25)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.5) pkg> add PyCall
Installing known registries into `~/.julia`
######################################################################## 100.0%
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed VersionParsing ─ v1.2.0
Installed Conda ────────── v1.4.1
Installed MacroTools ───── v0.5.5
Installed Parsers ──────── v1.0.10
Installed JSON ─────────── v0.21.0
Installed PyCall ───────── v1.91.4
Updating `~/.julia/environments/v1.5/Project.toml`
[438e738f] + PyCall v1.91.4
Updating `~/.julia/environments/v1.5/Manifest.toml`
[8f4d0f93] + Conda v1.4.1
[682c06a0] + JSON v0.21.0
[1914dd2f] + MacroTools v0.5.5
[69de0a69] + Parsers v1.0.10
[438e738f] + PyCall v1.91.4
[81def892] + VersionParsing v1.2.0
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[de0858da] + Printf
[9a3f8284] + Random
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[8dfed614] + Test
[4ec0a83e] + Unicode
Building Conda ─→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`
julia> ENV["PYTHON"]
ERROR: KeyError: key "PYTHON" not found
Stacktrace:
[1] (::Base.var"#477#478")(::String) at ./env.jl:79
[2] access_env at ./env.jl:43 [inlined]
[3] getindex(::Base.EnvDict, ::String) at ./env.jl:79
[4] top-level scope at REPL[2]:1
julia> using PyCall
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
julia> PyCall.python
"python3"
julia> PyCall.conda
false
julia> ENV["PYTHON"]=""
""
(@v1.5) pkg> build PyCall
Building Conda ─→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`
julia> using PyCall
julia> PyCall.conda
false
julia> PyCall.python
"python3"
If I exit out and reuse I’m the correct setting result:
julia> exit()
gregorylemieux@pop-os:~$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.1 (2020-08-25)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PyCall
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
julia> PyCall.python
"/home/gregorylemieux/.julia/conda/3/bin/python"
julia> PyCall.conda
true
So I guess this is me not understanding how to appropriately use using?
1 Like
Ah, I see what you mean.
This is the expected behavior.
If you do either import SomePackage or using SomePackage, and then you do Pkg.build("SomePackage"), any changes made during by Pkg.build will not be reflected during the same Julia session.
1 Like
Gotcha. I’m still curious why I was seeing the issue on my backed up version of julia. I’ll see if I can replicate the issue following your directions.
That said, thanks for all the help!