Julia recognizing installations as code

anistrope · July 6, 2023, 4:23am

Trying to follow the troubleshooting guide julia gives after PyPlot fails to use the matplotlib package. Specifically, I am following this part:

“PyCall is currently configured to use the Julia-specific Python distribution
installed by the Conda.jl package. To install the matplotlib module, you can
use pyimport_conda("matplotlib", PKG)”

But when I type pyimport_conda("matplotlib", PKG) in my Julia REPL (terminal), it gives:

ERROR: UndefVarError: pyimport_conda not defined
Stacktrace:
[1] top-level scope
@ REPL[7]:1

Like I was trying to define a variable. Pkg mode does not recognize pyimport_conda as a command either. So where is the guide talking about inputting this command?

Krastanov · July 6, 2023, 4:28am

Julia and its Pkg module have no (built-in) tools to work with python or conda. You need to import the (independent) package that provides this interoperability. From googling around, I assume that would be either using PyCall or using Conda.

anistrope · July 6, 2023, 4:56am

I tried this using the other parts of the guide. First things first PyCall does little except if it’s directed to a python distribution using matplotlib (I assume, the guide says so but I haven’t been able to do that yet). In fact I installed PyCall as you can tell by the first part of the troubleshooting message I quoted, it just failed to get matplotlib, which I tried to install with conda, which failed as well. As for ‘using Conda’, I went

‘using Conda’

‘Conda.add(“matplotlib”)’

It then gives:

Info: Running conda install -y matplotlib in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.

ResolvePackageNotFound:

conda==23.1.0

with a proper failed process message that would take up the entire screen.

Krastanov · July 6, 2023, 6:06am

I seem to be able to reproduce your second issue. Here is what I did (posted so that PyCall devs can debug)

shell> julia --project=. # to launch julia with a dedicated environment file Project.toml in the current folder

julia> ENV["PYTHON"] = "" # to ensure PyCall installs a separate Python binary instead of using your system one, so that it can manage it without messing up your system

pkg> add PyCall # to make PyCall available in this environment

pkg> build PyCall # to rebuild PyCall in case it has been already build with the system Python binary

julia> exit() # to quit julia

shell> julia --project=. # restart julia so that you can use the newly built PyCall

julia> using PyCall

julia> pyimport_conda("matplotlib", "matplotlib") # to import (and if necessary download) matplotlib

That last command fails with conda reporting it can not resolve. Conda.add fails the same way. Directly invoking ~/.julia/conda/.../bin/conda install matplotlib fails in the same way. My system’s conda does not seem to fail.

pyimport("numpy") works fine; pyimport_conda("qutip", "qutip") works fine; so it seems there is some problem specific to matplotlib and Conda.jl

Even using PyPlot fails the same way.

The stacktrace is below:

julia> using PyPlot
[ Info: Installing matplotlib via the Conda matplotlib package...
[ Info: Running `conda install -y matplotlib` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.
ResolvePackageNotFound:

conda==23.1.0

ERROR: InitError: failed process: Process(setenv(/home/stefan/.julia/conda/3/x86_64/bin/conda install -y matplotlib,[“PATH=/home/stefan/.juliaup/bin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl”, “DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus”, “XDG_SESSION_DESKTOP=gnome”, “XDG_SESSION_TYPE=wayland”, “SYSTEMD_EXEC_PID=1792”, “OPENBLAS_DEFAULT_NUM_THREADS=1”, “LC_TIME=en_GB.UTF-8”, “USER=stefan”, “LC_MEASUREMENT=en_GB.UTF-8”, “USERNAME=stefan”  …  “GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/7e4e215d_e19a_4c28_9620_90184f48ea81”, “GNOME_TERMINAL_SERVICE=:1.871”, “XDG_SESSION_CLASS=user”, “PWD=/tmp”, “DISPLAY=:0”, “GNOME_SETUP_DISPLAY=:1”, “MAIL=/var/spool/mail/stefan”, “CONDA_PREFIX=/home/stefan/.julia/conda/3/x86_64”, “OPENBLAS_MAIN_FREE=1”, “GNOME_KEYRING_CONTROL=/run/user/1000/keyring”]), ProcessExited(1)) [1]
Stacktrace:

[1] pipeline_error

@ ./process.jl:565 [inlined]

[2] run(::Cmd; wait::Bool)

@ Base ./process.jl:480

[3] run

@ ./process.jl:477 [inlined]

[4] runconda(args::Cmd, env::String)

@ Conda ~/.julia/packages/Conda/xMClC/src/Conda.jl:128

[5] add(pkg::String, env::String; channel::String, satisfied_skip_solve::Bool, args::Cmd)

@ Conda ~/.julia/packages/Conda/xMClC/src/Conda.jl:229

[6] add (repeats 2 times)

@ ~/.julia/packages/Conda/xMClC/src/Conda.jl:222 [inlined]

[7] pyimport_conda

@ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:721

[8] pyimport_conda

@ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:715 [inlined]

[9] init()

@ PyPlot ~/.julia/packages/PyPlot/H01LC/src/init.jl:173

[10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)

@ Base ./loading.jl:1115

[11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})

@ Base ./loading.jl:1061

[12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)

@ Base ./loading.jl:1506

[13] _require(pkg::Base.PkgId, env::String)

@ Base ./loading.jl:1783

[14] _require_prelocked(uuidkey::Base.PkgId, env::String)

@ Base ./loading.jl:1660

[15] macro expansion

@ ./loading.jl:1648 [inlined]

[16] macro expansion

@ ./lock.jl:267 [inlined]

[17] require(into::Module, mod::Symbol)

@ Base ./loading.jl:1611

during initialization of module PyPlot
caused by: PyError (PyImport_ImportModule
The Python package matplotlib could not be imported by pyimport. Usually this means

that you did not install matplotlib in the Python version being used by PyCall.
PyCall is currently configured to use the Julia-specific Python distribution

installed by the Conda.jl package.  To install the matplotlib module, you can

use pyimport_conda("matplotlib", PKG), where PKG is the Anaconda

package that contains the module matplotlib, or alternatively you can use the

Conda package directly (via using Conda followed by Conda.add etcetera).
Alternatively, if you want to use a different Python distribution on your

system, such as a system-wide Python (as opposed to the Julia-specific Python),

you can re-configure PyCall with that Python.   As explained in the PyCall

documentation, set ENV[“PYTHON”] to the path/name of the python executable

you want to use, run Pkg.build(“PyCall”), and re-launch Julia.
) <class ‘ModuleNotFoundError’>

ModuleNotFoundError(“No module named ‘matplotlib’”)
Stacktrace:

[1] pyimport(name::String)

@ PyCall ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:558

[2] pyimport_conda

@ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:716

[3] pyimport_conda

@ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:715 [inlined]

[4] init()

@ PyPlot ~/.julia/packages/PyPlot/H01LC/src/init.jl:173

[5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)

@ Base ./loading.jl:1115

[6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})

@ Base ./loading.jl:1061

[7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)

@ Base ./loading.jl:1506

[8] _require(pkg::Base.PkgId, env::String)

@ Base ./loading.jl:1783

[9] _require_prelocked(uuidkey::Base.PkgId, env::String)

@ Base ./loading.jl:1660

[10] macro expansion

@ ./loading.jl:1648 [inlined]

[11] macro expansion

@ ./lock.jl:267 [inlined]

[12] require(into::Module, mod::Symbol)

@ Base ./loading.jl:1611

@stevengj , pinging you as you seem to be most active PyCall / PyPlot maintainer. Any idea what is happening?

My versioninfo is:

julia> versioninfo()
Julia Version 1.9.1
Commit 147bdf428cd (2023-06-07 08:27 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × AMD Ryzen 9 7950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 32 on 32 virtual cores

Krastanov · July 6, 2023, 6:18am

I was able to fix it by:

Removing the Project.toml and Manifest.toml files I had
Removing ~/.julia/conda and ~/.julia/compiled/1.9
Starting julia --project=. for a fresh new environment without miscompiled previous attempts
ENV["PYTHON"] = "" then pkg> gc then pkg> add Conda then using Conda; Conda.add("matplotlib")

And it finally installed. Best I can tell, something got messed up on the first install and the build command failed to reset the various compiled caches so I had to purge them and start from scratch. This is definitely a bug specific to Conda.jl and matplotlib. @anistrope , if you can reliably reproduce it, the PyCall devs would love to have it reported so that they can fix it.

anistrope · July 6, 2023, 9:08am

Wow man. That actually worked. Thank you. I stayed up to 5 AM with this so I’m not in the mood to recreate that problem and stress about getting it right again until I wake up, but I’ll do my best to recount my process and what I think was the cause of it before I go to bed.

I agree that something was messed up on the first install. If I recall correctly this is the order that I installed things (and how)

Python
Julia (installed the tar.gz file on the website)
Then I tried to run the PyPlot command. This brought up the fact that PyCall (which was installed in the attempt to run this command) used the python I had installed in /usr/bin/python3. When you try to run PyPlot here, it should give you an error message saying that along with the fact that you do not have matplotlib (at least, not if you installed a python without this module). You may just think it’s an easy solution - install matplotlib into that version of python using the terminal - but it creates a whole separate folder and the python3 it was recognizing was an executable. Not sure how I could’ve made julia recognize python3 along with its modules.
So I tried to look up ways to get matplotlib and I found Anaconda. I Installed it, and in the process creating my own Anaconda3 folder.
I originally installed matplotlib using conda install matplotlib, which only put it in the Anaconda folder. When I tried to follow other parts of the guide, julia then made its own conda folder - which tried to make a matplotlib module but failed in the “finding environment” step.

So it should be something I did, my best guess is installing matplotlib on conda before Julia. Another interesting thing is no part of your guide mentions PyCall, even though the part of PyCall which messes this all up is matplotlib. The guide heavily implies that it’s a matter of where you install PyCall that matters, but I installed PyCall AFTER matplotlib in your guide. My plebian insights aside I will attach the images I still have of the first error message after PyPlot is unsuccessful lol.

Krastanov · July 6, 2023, 12:15pm

For what is worth, here are a couple not directly related suggestions, but nonetheless, suggestions that might make your setup a bit more robust. You might know of them, but just putting them here for other readers.

when using python (even if / especially if julia is not involved at all), use either python3 -m venv FOLDER_NAME or conda create --name ENVIRONMENT_NAME to create separate python environments independent from your system python. It helps with reproducibility, version mismatch, sysadmin, package installation, and protects your system python from modifications that would break it. Use pip / conda to install packages in that environment. Never use root.
when using python, very frequently use in your terminal the which command to see which python is currently ready to run, e.g. run which python to see whether you are running the system one or the one in the “virtual environment”
when using julia, use julia --project=FOLDER_NAME to start julia with a dedicated project environment (represented by Project.toml and Manifest.toml)
when using julia, consider using a version multiplexer like juliaup as a more convenient way to install and manage versions of julia

schlichtanders · July 8, 2023, 7:02am

I am running into the same issue and the hint it has something to do with the install makes a lot of sense

while on an OS with real user, everything is fine
on a docker with root user I get the mentioned error

So far I am unable to fix it on docker (I need to keep the root user because Github Actions require a root docker).
If someone knows a hint how to fix it on docker, it is highly appreciated.

schlichtanders · July 8, 2023, 2:23pm

I have looked into more details and for me the reason seems to be that numpy is installed before matplotlib (implicit when using PyCall). The other way around, or both at once works seamlessly, but numpy first then matplitlib fails.

see ResolvePackageNotFound: - conda==23.1.0 · Issue #238 · JuliaPy/Conda.jl · GitHub
I hope this can be fixed