Conda Not installing Matplotlib for PyPlot

Perrin_Meyer · March 30, 2023, 1:02am

I started with a fresh install of Julia 1.9.0-rc1 (I wiped .julia clean)

PyCall Conda does not seem to be installing matplotlib correctly. This has worked for me many times before on many different machines…The culprit seems to be

ResolvePackageNotFound: 
  - conda==23.1.0

which is strange, since If I look in .julia, I see two conda Versions

perrin@perrin76:~/.julia/conda/3/x86_64/pkgs$ ls con*
conda-23.1.0-py310hff52083_0.conda  conda-23.3.0-py310hff52083_0.conda  conda-package-handling-2.0.2-pyh38be061_0.conda  conda-package-streaming-0.7.0-pyhd8ed1ab_1.conda

conda-23.1.0-py310hff52083_0:
bin  condabin  etc  info  lib  shell

conda-23.3.0-py310hff52083_0:
bin  condabin  etc  info  lib  shell

The full Dump:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-rc1 (2023-03-07)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> ENV["PYTHON"]=""
""

(@v1.9) pkg> build PyCall
    Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/e32a90da027ca45d84678b826fffd3110bb3fc90/build.log`
    Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/62f417f6ad727987c755549e9cd88c46578da562/build.log`

(@v1.9) pkg> add PyPlot
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.9/Project.toml`
  No Changes to `~/.julia/environments/v1.9/Manifest.toml`

julia> using PyPlot
[ Info: Installing matplotlib via the Conda matplotlib package...
[ Info: Running `conda install -y matplotlib` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.

ResolvePackageNotFound: 
  - conda==23.1.0

ERROR: InitError: failed process: Process(setenv(`/home/perrin/.julia/conda/3/x86_64/bin/conda install -y matplotlib`,["OPENBLAS_MAIN_FREE=1", "PYTHONIOENCODING=UTF-8", "PATH=/home/perrin/.local/bin:/home/perrin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/perrin/.local/bin:/home/perrin/bin:", "QT_ACCESSIBILITY=1", "LD_LIBRARY_PATH=:/opt/OpenBLAS/lib/:/home/perrin/FMM3D/lib/:/home/perrin/fmm3dbie/lib/:", "INVOCATION_ID=f87f231ef0cc4e689efd514b7525fbf4", "IM_CONFIG_PHASE=1", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus", "XDG_CURRENT_DESKTOP=ubuntu:GNOME", "LANG=en_US.UTF-8"  …  "MANAGERPID=1693", "SESSION_MANAGER=local/perrin76:@/tmp/.ICE-unix/1912,unix/perrin76:/tmp/.ICE-unix/1912", "XMODIFIERS=@im=ibus", "HOME=/home/perrin", "TERM=xterm-256color", "WINDOWPATH=2", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:", "COLORTERM=truecolor", "SPIRAL_HOME=/home/perrin/spiral-software", "VTE_VERSION=6003"]), ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] #run#774
    @ ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] runconda
    @ ~/.julia/packages/Conda/kOnIE/src/Conda.jl:128
  [5] #add#1
    @ ~/.julia/packages/Conda/kOnIE/src/Conda.jl:224
  [6] add (repeats 2 times)
    @ ~/.julia/packages/Conda/kOnIE/src/Conda.jl:222 [inlined]
  [7] pyimport_conda
    @ ~/.julia/packages/PyCall/twYvK/src/PyCall.jl:721
  [8] pyimport_conda
    @ ~/.julia/packages/PyCall/twYvK/src/PyCall.jl:715 [inlined]
  [9] __init__
    @ ~/.julia/packages/PyPlot/H01LC/src/init.jl:173
 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1074
 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1020
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1465
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1742
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1619
 [15] macro expansion
    @ ./loading.jl:1607 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1570
during initialization of module PyPlot

caused by: PyError (PyImport_ImportModule

The Python package matplotlib could not be imported by pyimport. Usually this means
that you did not install matplotlib in the Python version being used by PyCall.

PyCall is currently configured to use the Julia-specific Python distribution
installed by the Conda.jl package.  To install the matplotlib module, you can
use `pyimport_conda("matplotlib", PKG)`, where PKG is the Anaconda
package that contains the module matplotlib, or alternatively you can use the
Conda package directly (via `using Conda` followed by `Conda.add` etcetera).

Alternatively, if you want to use a different Python distribution on your
system, such as a system-wide Python (as opposed to the Julia-specific Python),
you can re-configure PyCall with that Python.   As explained in the PyCall
documentation, set ENV["PYTHON"] to the path/name of the python executable
you want to use, run Pkg.build("PyCall"), and re-launch Julia.

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'matplotlib'")

Stacktrace:
  [1] pyimport
    @ ~/.julia/packages/PyCall/twYvK/src/PyCall.jl:558
  [2] pyimport_conda
    @ ~/.julia/packages/PyCall/twYvK/src/PyCall.jl:716
  [3] pyimport_conda
    @ ~/.julia/packages/PyCall/twYvK/src/PyCall.jl:715 [inlined]
  [4] __init__
    @ ~/.julia/packages/PyPlot/H01LC/src/init.jl:173
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1074
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1020
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1465
  [8] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1742
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1619
 [10] macro expansion
    @ ./loading.jl:1607 [inlined]
 [11] macro expansion
    @ ./lock.jl:267 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1570

julia>

Ssz990220 · March 30, 2023, 8:39am

Hi, it seems like I solved this problem by downgrading the conda to 23.1.0.

Julia creates its conda env at ~/.julia/conda/3/x86_64/. So what I did was activate this env by conda activate ~/.julia/conda/3/x86_64/. Then manually downgrade the conda version by conda install conda=23.1.0.

After downgrading the conda, I install the matplotlib package by conda install matplotlib.

I hope this works for you.

mateuszbaran · March 30, 2023, 2:23pm

Hi! We had a similar issue in CI and decided to switch from PyPlot.jl to PythonPlot.jl (see Using PythonPlot instead of PyPlot in tests by sethaxen · Pull Request #585 · JuliaManifolds/Manifolds.jl · GitHub).

nirong · April 9, 2023, 8:43am

I had a similar issue when I ran "Conda.add(“geopandas”) " in julia. Can you tell me where did you run “conda activate ~/.julia/conda/3/x86_64/” and “conda install conda=23.1.0” .

Ssz990220 · April 9, 2023, 8:54am

I think you should first find the exact location where the conda of julia is installed.
In my case, I am using a Windows PC, the location of the conda env is C:\Users\USERNAME\.julia\conda\3\x86_64. So I activate the env with command conda activate C:\Users\USERNAME\.julia\conda\3\x86_64\
I am not so sure about linux, but it should be somewhere alike.

I think it does not matter where you run the command, as long as you get the absolute path correct.

nirong · April 9, 2023, 9:19am

Thanks for your advice. I run “conda activate C:/Users/DELL/.julia/conda/3/x86_64/” and “conda install conda=23.1.0” in Anaconda Promt, and conda=23.1.0 successfuly downloaded. But the error appeared in julia still exsists. In julia, I run " Pkg.activate(joinpath(“E:/GeoRDD.jl-master/examples/NYC/”)) " before run " Conda.add(“geopandas”) " , is this the reason that caused this problem? Should there a “conda” folder in “E:/GeoRDD.jl-master/examples/NYC/”?

Ssz990220 · April 10, 2023, 1:55am

I am not pretty sure about your problem.
And I don’t think there should be a ‘conda’ folder in “E:/GeoRDD.jl-master/examples/NYC/”.

In my case, I installed the matplotlib with conda install matplotlilb in conda instead of julia conda. So I think you should try this first. It is worth a shot, though I am not certain that it will fix the problem.

May you provide the full Stacktrace? That way we can better understand the issue you run into.

nirong · April 10, 2023, 4:15am

Thanks for your reply. Here is error.

Here is the original code.

Ssz990220 · April 10, 2023, 6:04am

It might be a python package dependency issue rather than a julia issue, as it is suggested by the traceback module 'geopandas' has no attribute 'plotting'. I have no clue about what may cause the problem, but I suggest to first check your dependency in the conda env.

BTW, I failed to reproduce your problem on my end, which indicates that it might be a machine-specific or env-specific issue. I built my env by the following steps:

create a new julia project and activate it
install conda in julia
manually downgrade conda and install matplotlib
manually install geopandas in the conda env with conda command line
using PyCall in julia to executed these line which potentially cause the problem
And it works just fine for me.

(MyJuliaEnv) pkg> st
Status `C:\Users\USERNAME\Project\MyJuliaEnv\Project.toml`
  [b4f34e82] Distances v0.10.8
⌃ [31c24e10] Distributions v0.25.86
  [c8e1da08] IterTools v1.4.0
  [b8a86587] NearestNeighbors v0.4.13
  [438e738f] PyCall v1.95.1
  [d330b81b] PyPlot v2.11.1
  [189a3867] Reexport v1.2.2
  [c946c3f1] SCS v1.1.3
  [90137ffa] StaticArrays v1.5.19
Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> using PyCall

julia> py"""
       import matplotlib.pyplot as plt
       import pandas as pd
       import geopandas as gpd
       plot_dataframe = gpd.plotting.plot_dataframe
       import numpy as np
       import itertools
       """

julia>

nirong · April 10, 2023, 7:15am

Thanks for your reply. I test the code which prompts error in notebook solely, and there is no error. However I run the whole code, the error still exists.

Ssz990220 · April 10, 2023, 7:51am

It seems like a dependency issue with GeoPandas. Perhaps reinstalling the Conda environment will resolve the problem.

nirong · April 10, 2023, 8:08am

I am new to julia. Can you show a example for " reinstalling the Conda environment"? Thanks a lot.

Ssz990220 · April 10, 2023, 8:25am

Apology for the possible misunderstanding.

By “reinstalling the conda environment”, I did not mean some action in Julia. Instead, I am referring to taking actions with conda prompt. Like:

activate the conda env as before by conda activate C:\Users\DELL\.julia\conda\3\x86_64\
remove or install some python package with conda install XXX or conda remove XXX, etc…

You can also test if you setup everything correctly without julia by:

activate conda env
start a python REPL by python, you can check if you successfully activate the python by

import sys
print(sys.executable)

which should give you C:\Users\DELL.julia\conda\3\x86_64.
3. run the python lines in this REPL

import matplotlib.pyplot as plt
import pandas as pd
import geopandas as gpd
plot_dataframe = gpd.plotting.plot_dataframe
import numpy as np
import itertools

If no error is encountered here, your Julia code should just run ok.

My take is, the problem you ran into is a complete Python dependency issue, thus it can be resolved with conda prompt. So no need to worry about Julia for the moment. And maybe you should find relevant discussion in a Python forum, which obviously has more Python-related resources than what you can find here.

Best

nirong · April 10, 2023, 8:36am

I got what you mean. Thanks again for your detailed explanation.