How, I ran into some strange behavior with Julia.
I have a software package that works like this
- Julia user code starts (via
dlopen) a C library that instantiates an embedded Python
- Python user code does the same
Everything worked just a couple of days ago.
However, I do not quite understand why, it broke: for some strange reason when the Julia user code is used, the path to libpython is completely wrong (it points to a Homebrew Python, in Ubuntu instead of the active Conda environment), and everything breaks.
When the user code is in Python, it works correctly: libpython corresponds to the actual Python executable.
While trying to find out, why this happens, I have tried to install PyCall.jl and got the following:
julia> using PyCall
│ Package PyCall not found, but a package named PyCall is available from a registry.
│ Install package?
│ (um02-open-interfaces) pkg> add PyCall
└ (y/n/o) [y]: y
Resolving package versions...
Updating `~/Work/um02-open-interfaces/Project.toml`
[438e738f] + PyCall v1.96.4
Updating `~/Work/um02-open-interfaces/Manifest.toml`
[8f4d0f93] + Conda v1.10.2
[438e738f] + PyCall v1.96.4
[81def892] + VersionParsing v1.3.0
Precompiling PyCall...
Info Given PyCall was explicitly requested, output will be shown live
ERROR: LoadError: could not load library "/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0"
/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory. Please run `Pkg.build("PyCall")` if your Python build has changed
My actual Conda Python is 3.13.2, not 3.11. Question why Julia memorizes somewhere the path to libpython? And, to the problem above, why Julia somehow breaks the search of the correct libpython inside a dynamically loaded C library?
I am on Julia 1.11.5, Ubuntu 22.04.
It’s the specific package that you’re using which handles how python is searched, reading its documentation may be helpful: PyCall.jl/README.md at 2f600fbebee50ab0672e153455e3c0fda1694fba · JuliaPy/PyCall.jl · GitHub
Are you running Julia from the terminal, or from a VSCode terminal?
VSCode changes the python path if you ever installed some Python plugins using VSCode.
I am in the terminal. The weird thing is that when the (incorrect) libpython is loaded, subsequent import sys; print(sys.executable) shows the correct python (the one which gets invoked in the terminal as a command). Everything actually worked, because I had it working. Something got updated (probably, the homebrew packages), and it does not work anymore 
The version that pyCall.jl tries to use (Python 3.11) is not there at all. It was there before, but sometime in the past the conda environment was recreated and uses Python 3.13 now. That’s why I suspect some form of caching happening.
Did you read the link I shared which tells you what you need to do to reset the path to python?
2 Likes
Right. PyCall saves the path to libpython. You need to rebuild PyCall if you want to use a different libpython, or if your Python moves.
Hi, sorry for the late reply, I am in another time zone.
Thank you for the link, I have just tried to set the PYTHON variable before installing PyCall (by the way, I am not reinstalling it, at least not in the current Julia environment), I receive the exactly same error where PyCall is looking for a verion of Python that is just not there (I use 3.13, not 3.11 that it is looking for):
um02-open-interfaces on julia-ivp [$!?] via 🅒 um02-open-interfaces …
➜ which python
/home/dima/sw/miniforge3/envs/um02-open-interfaces/bin/python
um02-open-interfaces on julia-ivp [$!?] via 🅒 um02-open-interfaces …
➜ PYTHON=/home/dima/sw/miniforge3/envs/um02-open-interfaces/bin/python julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.5 (2025-04-14)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(um02-open-interfaces) pkg> add PyCall
Resolving package versions...
No Changes to `~/Work/um02-open-interfaces/Project.toml`
No Changes to `~/Work/um02-open-interfaces/Manifest.toml`
Precompiling project...
✗ PyCall
✗ SciMLBase → SciMLBasePyCallExt
0 dependencies successfully precompiled in 6 seconds. 266 already precompiled.
2 dependencies errored.
For a report of the errors see `julia> err`. To retry use `pkg> precompile`
julia> err
PkgPrecompileError: The following 1 direct dependency failed to precompile:
PyCall
Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to "/home/dima/.julia/compiled/v1.11/PyCall/jl_Qptcah".
ERROR: LoadError: could not load library "/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0"
/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory. Please run `Pkg.build("PyCall")` if your Python build has changed
Stacktrace:
[1] error(::String, ::String)
@ Base ./error.jl:44
[2] top-level scope
@ ~/.julia/packages/PyCall/1gn3u/src/startup.jl:51
[3] include(mod::Module, _path::String)
@ Base ./Base.jl:557
[4] include(x::String)
@ PyCall ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:1
[5] top-level scope
@ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:38
[6] include
@ ./Base.jl:557 [inlined]
[7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base ./loading.jl:2881
[8] top-level scope
@ stdin:6
in expression starting at /home/dima/.julia/packages/PyCall/1gn3u/src/startup.jl:41
in expression starting at /home/dima/.julia/packages/PyCall/1gn3u/src/PyCall.jl:1
in expression starting at stdin:6
caused by: could not load library "/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0"
/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:120
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:119
[3] top-level scope
@ ~/.julia/packages/PyCall/1gn3u/src/startup.jl:48
[4] include(mod::Module, _path::String)
@ Base ./Base.jl:557
[5] include(x::String)
@ PyCall ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:1
[6] top-level scope
@ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:38
[7] include
@ ./Base.jl:557 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base ./loading.jl:2881
[9] top-level scope
@ stdin:6
On a side note, before that the PYTHON env variable was not set at all, but PyCall was still looking for the libpython inside the actual Conda environment that I use.
Right. PyCall saves the path to libpython. You need to rebuild PyCall if you want to use a different libpython, or if your Python moves.
This I can understand: if there is linkage to libpython, of course.
However, I am adding PyCall to the Julia environment, not reinstalling it. Or do you mean, that the path is saved somewhere globally, across multiple environments?
I don’t see evidences that you ran the command Pkg.build("PyCall"), as explained in both the link I shared and the error message you posted.
Ah, OK, I did not understand before that > add PyCall and Pkg.build("PyCall") are two different things. Yes, by invoking PYTHON=path-here julia and using the Pkg.build command, I was able to install it. Nevertheless, the error message above about a completely wrong version of Python that is not there, is still very cryptic.
And my main problem was that when starting with user code in Julia and then using a C library that starts an embedded Python, the wrong libpython was loaded (and this does not happen with user code in C or Python). I actually found the source of the error. For some reason, I had wrong python3 priority in PATH (python was correct), and this was the influence.
Thank you everybody!
This part was not clear:
ERROR: LoadError: could not load library "/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0"
/home/dima/.conda/envs/um02-open-interfaces/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory. Please run `Pkg.build("PyCall")` if your Python build has changed
considering that I have never had this package installed in the used Julia environment. Why was it looking for Python 3.11 in the Conda environment with Python 3.13?
You must have installed PyCall using that Conda environment at some point and forgot about it.