Unable to add PyCall to any environment

peter.wolf · July 7, 2022, 12:52am

I added an issue to PyCall.jl but I think it is more likely that something has become messed up on my laptop

PyCall broken with 1.7.3?

opened 11:39PM - 05 Jul 22 UTC

I am hoping my environment is messed up somehow I just start Julia, 'using Py…Call' and y to 'add PyCall' I get 'ERROR: LoadError: PyCall not properly installed. Please run Pkg.build("PyCall")' See below for the requested and failed Pkg.build("PyCall") The log files are here https://drive.google.com/file/d/1pXBCoMkFoQu9bOEhGvu51ntcgbAHs0-v/view?usp=sharing https://drive.google.com/file/d/10uqQc-4ih_1A-uSvZPT4l404pJJdjq_e/view?usp=sharing ` (base) C02GC228MD6T:~ peter.wolf$ julia ...Version 1.7.3 (2022-05-06) ... julia> using PyCall │ Package PyCall not found, but a package named PyCall is available from a │ registry. │ Install package? │ (@v1.7) pkg> add PyCall └ (y/n) [y]: y Updating registry at `~/.julia/registries/General.toml` Resolving package versions... Updating `~/.julia/environments/v1.7/Project.toml` [438e738f] + PyCall v1.93.1 Updating `~/.julia/environments/v1.7/Manifest.toml` [8f4d0f93] + Conda v1.7.0 [682c06a0] + JSON v0.21.3 [1914dd2f] + MacroTools v0.5.9 [69de0a69] + Parsers v2.3.2 [438e738f] + PyCall v1.93.1 [81def892] + VersionParsing v1.3.0 [37e2e46d] + LinearAlgebra [a63ad114] + Mmap [e66e0078] + CompilerSupportLibraries_jll [4536629a] + OpenBLAS_jll [8e850b90] + libblastrampoline_jll [ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] ERROR: LoadError: PyCall not properly installed. Please run Pkg.build("PyCall") Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] top-level scope @ ~/.julia/packages/PyCall/7a7w0/src/startup.jl:44 [3] include(mod::Module, _path::String) @ Base ./Base.jl:418 [4] include(x::String) @ PyCall ~/.julia/packages/PyCall/7a7w0/src/PyCall.jl:1 [5] top-level scope @ ~/.julia/packages/PyCall/7a7w0/src/PyCall.jl:38 [6] include @ ./Base.jl:418 [inlined] [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing) @ Base ./loading.jl:1318 [8] top-level scope @ none:1 [9] eval @ ./boot.jl:373 [inlined] [10] eval(x::Expr) @ Base.MainInclude ./client.jl:453 [11] top-level scope @ none:1 in expression starting at /Users/peter.wolf/.julia/packages/PyCall/7a7w0/src/startup.jl:41 in expression starting at /Users/peter.wolf/.julia/packages/PyCall/7a7w0/src/PyCall.jl:1 ERROR: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /Users/peter.wolf/.julia/compiled/v1.7/PyCall/jl_ypJJQ7. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool) @ Base ./loading.jl:1466 [3] compilecache(pkg::Base.PkgId, path::String) @ Base ./loading.jl:1410 [4] _require(pkg::Base.PkgId) @ Base ./loading.jl:1120 [5] require(uuidkey::Base.PkgId) @ Base ./loading.jl:1013 [6] require(into::Module, mod::Symbol) @ Base ./loading.jl:997 ` ` julia> using Pkg julia> Pkg.build("PyCall") Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6e47d11ea2776bc5627421d59cdcc1296c058071/build.log` Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/1fc929f47d7c151c839c5fc1375929766fb8edcc/build.log` ERROR: Error building `PyCall`, showing the last 100 of log: transformers -> huggingface-hub[version='<1.0,>=0.1.0'] huggingface-hub Package responses conflicts for: datasets -> responses[version='<0.19'] responses Package importlib-metadata conflicts for: transformers -> importlib_metadata -> importlib-metadata[version='>=1.1.3,<1.1.4.0a0|>=1.6.0,<1.6.1.0a0|>=1.6.1,<1.6.2.0a0|>=1.7.0,<1.7.1.0a0|>=2.0.0,<2.0.1.0a0|>=3.10.0,<3.10.1.0a0|>=4.11.3,<4.11.4.0a0|>=4.8.2,<4.8.3.0a0|>=4.8.1,<4.8.2.0a0|>=4.6.4,<4.6.5.0a0|>=3.7.3,<3.7.4.0a0'] sacremoses -> click -> importlib-metadata datasets -> importlib_metadata -> importlib-metadata[version='>=1.1.3,<1.1.4.0a0|>=1.6.0,<1.6.1.0a0|>=1.6.1,<1.6.2.0a0|>=1.7.0,<1.7.1.0a0|>=2.0.0,<2.0.1.0a0|>=3.10.0,<3.10.1.0a0|>=4.11.3,<4.11.4.0a0|>=4.8.2,<4.8.3.0a0|>=4.8.1,<4.8.2.0a0|>=4.6.4,<4.6.5.0a0|>=3.7.3,<3.7.4.0a0'] click -> importlib-metadata Package cffi conflicts for: pytorch -> cffi transformers -> pytorch -> cffi conda-content-trust -> cryptography -> cffi[version='!=1.11.3,>=1.8|>=1.12|>=1.8|>=1.7'] brotlipy -> cffi[version='>=1.0.0'] soundfile -> cffi[version='>=1.0'] cryptography -> cffi[version='!=1.11.3,>=1.8|>=1.12|>=1.8|>=1.7'] cffi pyopenssl -> cryptography[version='>=35.0'] -> cffi[version='!=1.11.3,>=1.8|>=1.12|>=1.8|>=1.7'] urllib3 -> brotlipy[version='>=0.6.0'] -> cffi[version='!=1.11.3,>=1.8|>=1.0.0|>=1.12|>=1.8|>=1.7'] Package multiprocess conflicts for: datasets -> multiprocess multiprocess Package ninja-base conflicts for: pytorch -> ninja -> ninja-base==1.10.2=haf03e11_5 ninja-base ninja -> ninja-base==1.10.2=haf03e11_5 Package sacremoses conflicts for: transformers -> sacremoses sacremoses Package async-timeout conflicts for: aiohttp -> async-timeout[version='<4.0,>=3.0|>=3.0,<4.0|>=4.0.0a3,<5.0|>=1.2,<3.0|>=1.2.0|<5.0,>=4.0.0a3'] datasets -> aiohttp -> async-timeout[version='<4.0,>=3.0|>=3.0,<4.0|>=4.0.0a3,<5.0|>=1.2,<3.0|>=1.2.0|<5.0,>=4.0.0a3'] async-timeout Package zstd conflicts for: pyarrow -> arrow-cpp[version='>=4.0.1,<4.0.2.0a0'] -> zstd[version='>=1.3.3,<1.3.4.0a0|>=1.3.7,<1.3.8.0a0|>=1.4.5,<1.5.0a0|>=1.4.9,<1.5.0a0'] conda-package-handling -> libarchive[version='>=3.3.3'] -> zstd[version='>=1.3.3,<1.3.4.0a0|>=1.3.7,<1.3.8.0a0|>=1.4.4,<1.5.0a0|>=1.5.0,<1.6.0a0|>=1.5.2,<1.6.0a0'] Package futures conflicts for: pyarrow -> futures conda[version='>=4.12.0'] -> conda-package-handling[version='>=1.3.0'] -> futures conda-package-handling -> futures Package joblib conflicts for: joblib transformers -> sacremoses -> joblib sacremoses -> joblib Package pyarrow conflicts for: pyarrow datasets -> pyarrow[version='>=0.17.1|>=5.0.0'] Package pandas conflicts for: pyarrow -> pandas datasets -> pandas pandas Package pyparsing conflicts for: packaging -> pyparsing[version='!=3.0.5,>=2.0.2|>=2.0.2|>=2.0.2,!=3.0.5'] pyparsing huggingface-hub -> packaging[version='>=20.9'] -> pyparsing[version='!=3.0.5,>=2.0.2|>=2.0.2|>=2.0.2,!=3.0.5'] datasets -> packaging -> pyparsing[version='!=3.0.5,>=2.0.2|>=2.0.2|>=2.0.2,!=3.0.5'] transformers -> packaging[version='>=20.0'] -> pyparsing[version='!=3.0.5,>=2.0.2|>=2.0.2|>=2.0.2,!=3.0.5'] Package zope.interface conflicts for: attrs -> zope.interface aiohttp -> attrs[version='>=17.3.0'] -> zope.interface ┌ Info: Using the Python distribution in the Conda package by default. └ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall"). [ Info: Running `conda install -y numpy` in root environment ERROR: LoadError: failed process: Process(setenv(`/Users/peter.wolf/miniconda3/envs/phinder/bin/conda install -y numpy`,["XPC_FLAGS=0x0", "_CE_M=", "PATH=/Users/peter.wolf/opt/miniconda3/bin:/Users/peter.wolf/opt/miniconda3/condabin:/Users/peter.wolf/.cargo/bin:/Applications/Julia-1.7.app/Contents/Resources/julia/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", "PWD=/Users/peter.wolf", "XPC_SERVICE_NAME=0", "TERM_PROGRAM=Apple_Terminal", "SHELL=/bin/bash", "MAVEN_OPTS=-Xmx1024m", "__CF_USER_TEXT_ENCODING=0x1F7:0x0:0x0", "OPENBLAS_NUM_THREADS=8" … "_CE_CONDA=", "USER=peter.wolf", "JULIA_EDITOR=emacsclient", "TERM=xterm-256color", "HOME=/Users/peter.wolf", "TERM_PROGRAM_VERSION=440", "JULIA_HOME=/Applications/Julia-1.7.app/Contents/Resources/julia", "JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home", "OPENBLAS_MAIN_FREE=1", "PYTHONIOENCODING=UTF-8"]), ProcessExited(1)) [1] Stacktrace: [1] pipeline_error @ ./process.jl:540 [inlined] [2] run(::Cmd; wait::Bool) @ Base ./process.jl:455 [3] run @ ./process.jl:453 [inlined] [4] runconda(args::Cmd, env::String) @ Conda ~/.julia/packages/Conda/x2UxR/src/Conda.jl:128 [5] add(pkg::String, env::String; channel::String) @ Conda ~/.julia/packages/Conda/x2UxR/src/Conda.jl:222 [6] add (repeats 2 times) @ ~/.julia/packages/Conda/x2UxR/src/Conda.jl:221 [inlined] [7] top-level scope @ ~/.julia/packages/PyCall/7a7w0/deps/build.jl:79 [8] include(fname::String) @ Base.MainInclude ./client.jl:451 [9] top-level scope @ none:5 in expression starting at /Users/peter.wolf/.julia/packages/PyCall/7a7w0/deps/build.jl:43 Full log at /Users/peter.wolf/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/1fc929f47d7c151c839c5fc1375929766fb8edcc/build.log Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68 [2] (::Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec})() @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:962 [3] withenv(::Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, ::Pair{String, String}, ::Vararg{Pair{String}}) @ Base ./env.jl:172 [4] (::Pkg.Operations.var"#99#103"{String, Bool, Bool, Bool, Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec})() @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1506 [5] with_temp_env(fn::Pkg.Operations.var"#99#103"{String, Bool, Bool, Bool, Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec}, temp_env::String) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1390 [6] (::Pkg.Operations.var"#98#102"{Bool, Bool, Bool, Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String})(tmp::String) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1469 [7] mktempdir(fn::Pkg.Operations.var"#98#102"{Bool, Bool, Bool, Pkg.Operations.var"#62#67"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String}, parent::String; prefix::String) @ Base.Filesystem ./file.jl:750 [8] mktempdir(fn::Function, parent::String) (repeats 2 times) @ Base.Filesystem ./file.jl:748 [9] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project; force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1435 [10] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1432 [11] build_versions(ctx::Pkg.Types.Context, uuids::Set{Base.UUID}; verbose::Bool) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:943 [12] build(ctx::Pkg.Types.Context, uuids::Set{Base.UUID}, verbose::Bool) @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:822 [13] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}}) @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:992 [14] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:149 [15] build(pkgs::Vector{Pkg.Types.PackageSpec}) @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:144 [16] #build#99 @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined] [17] build @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined] [18] #build#98 @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined] [19] build(pkg::String) @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [20] top-level scope @ REPL[3]:1 `

Basically, if I try to add PyCall to any environment, it fails with

ERROR: LoadError: PyCall not properly installed. Please run Pkg.build("PyCall")

and when I build it I get

Pkg.build("PyCall").
[ Info: Running `conda install -y numpy` in root environment
ERROR: LoadError: failed process: Process(setenv(`/Users/peter.wolf/miniconda3/envs/phinder/bin/conda install -y numpy` ,["XPC_FLAGS=0x0", "_CE_M=",

Looking for things to check. How could I have messed up the built in Python? How does PyCall find the built in Python?

Thanks in advance
Peter

Note that I am using the builtin Python

julia>ENV["PYTHON"]
""

jd-foster · July 7, 2022, 2:36am

I think you can isolate this error. Can you try

import Pkg
ENV["PYTHON"] = ""
Pkg.add("Conda")
import Conda
Conda.add("numpy")

?

peter.wolf · July 7, 2022, 3:31pm

Thank you JD. Full log here

peter.wolf · July 7, 2022, 3:40pm

If I understand this error, Conda.add(“numpy”) is failing because the Python version is inconsistent with several installed libraries

What Python does PyCall use by default? I thought the package contained a built-in libpython

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - pypi/pypi::datasets==2.1.0=pypi_0

Specifications:

  - brotlipy -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0']
  - pyarrow -> python[version='>=3.10,<3.11.0a0|>=3.7.1,<3.8.0a0']
  - pytorch -> python[version='>=3.5,<3.6.0a0']
  - wheel -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']

Your python: python=3.9

peter.wolf · July 7, 2022, 6:26pm

I also notice that Conda.jl is using an executable from one of my miniconda environments. How/why is it doing that? Phinder is not a currently active environment. Where is this information stored?

Process(setenv(/Users/peter.wolf/miniconda3/envs/phinder/bin/conda install -y numpy,...

peter.wolf · July 7, 2022, 6:44pm

Ah ha! Fixed it by removing pypi/pypi::datasets==2.1.0=pypi_0

Conda.rm("datasets")

I still don’t understand how Conda.jl and PyCall.jl choose their environment. But apparently I fixed the chosen one.

jd-foster · July 8, 2022, 1:04am

Glad you got it, my next recommendation was removing the conflicting packages.

Yes, I was also trying to work out why it was choosing this miniconda environment. You can check and/or set the default value using Conda.ROOTENV.
See also Using a pre-existing Conda installation

By the way, CondaPkg.jl with PythonCall.jl is a more nuanced alternative to managing conda environments while calling python that’s work for me in the past.