Hi All
trying out julia and Pluto for the first time. I “thought” was doing everything correctly but obviously not. I get
"The Python package test_ib_60_sec.py could not be imported by pyimport."
and
ModuleNotFoundError("No module named 'test_ib_60_sec'")
from the following code
using Pycall
ENV["PYTHON"]="/usr/bin/python3/"
np = pyimport("numpy")
py_script = pyimport("test_ib_60_sec.py")
np.array([1,2,3,4]) works fine so that’s ok
I am running the latest Pluto ( though I can’t see how to find that out) and julia 1.7.0 ( both fresh install)
I used the internal PKG in Pluto to add Pycall
The Pluto notebook AND the python script are in the same directory AND I checked the script name.
the Pluto notebook is “running in the same directory” according to pwd()
Any help would be appreciated.
Did you re-build PyCall after setting ENV["PYTHON"]? See here for instructions.
Edit: see also this note on adding the current directory to the Python module search path.
1 Like
@stillyslalom thanks for the link. I am VERY confused. As Pluto now has it’s own mechanism for Pkg management I have NO clue how to rebuild PyCall. I read through the instructions in the link you sent but to me it’s not apparent how to do this?
here’s what I tried
julia> import Pkg
julia> Pkg.build("PyCall")
ERROR: The following package names could not be resolved:
* PyCall (not found in project or manifest)
In Pluto the Using PyCall has a check mark by it and it says I am running PyCall 1.92.5
ALSO
thanks for the extra link regarding adding current directory to Python module search path, really helps
added
pushfirst!(PyVector(pyimport("sys")."path"), "")
to the top of my Pluto Notebook code
Thanks
Note that you don’t need to rebuild PyCall in Pluto. You can do it in the Julia REPL or anywhere else — you only need to do it once, to tell PyCall which python you want it to use, and it will remember it.
e.g., in the REPL:
julia> ENV["PYTHON"]="python3" # assumes python3 is in your system PATH
pkg> build PyCall
(Note that you get the pkg> prompt in the REPL by typing ].)
This looks wrong. PYTHON should be the pathname of the python executable you want, whereas the trailing / here makes it look like this is a directory?
Note also that if the python you want is in the PATH (i.e. can you run python3 from a shell without typing a pathname?) then you can just set ENV["PYTHON"]="python3". Note also that the system python3 is the default for PyCall on everything but Mac and Windows (where PyCall installs its own Python distro via Conda.jl).
You can check which python your PyCall is currently configured to use by looking at
using PyCall
PyCall.python
I would look at this first before mucking with the configuration — it could be that it is already what you want?
1 Like
Hi @stevengj
thanks for helping. I did try to rebuild PyCall in julia but got this
(@v1.7) pkg>
(@v1.7) pkg> build Pycall
ERROR: The following package names could not be resolved:
* Pycall (not found in project or manifest)
Note that I used the latest Package manager in Pluto New Pluto Pkg manager and so am a TAD wary of doing anything outside it as I don’t want to turn it off. I can use PyCall in my Pluto notebook ( see below for Pycall.python) so it’s in Pluto but I don’t know how to rebuild it
As regards the call to PyCall.python it returns
"python3"
I can run python3 from a shell $python3 yields python 3.8.10 which is great.
so I’ll quit messing around with it for the time being.
Julia’s modules are case-sensitive, so you need to use PyCall, not Pycall.
I did double check that but I don’t think that’s the problem. I can use PyCall in my Pluto notebook because I added it via the new Pluto Pkg manager. It doesn’t seem to be in the julia manifest.
I check that PyCall is working by calling NUMPY and loading up a numpy array which works fine
dave@deepthought:~$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> import Pkg
julia> Pkg.build("PyCall")
ERROR: The following package names could not be resolved:
* PyCall (not found in project or manifest)
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
[2] ensure_resolved(manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
@ Pkg.Types ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/Types.jl:915
[3] ensure_resolved
@ ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/Types.jl:878 [inlined]
[4] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:991
[5] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
[6] build(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
[7] #build#99
@ ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined]
[8] build
@ ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined]
[9] #build#98
@ ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
[10] build(pkg::String)
@ Pkg.API ~/julia-1.7.0/share/julia/stdlib/v1.7/Pkg/src/API.jl:141
[11] top-level scope
@ REPL[1]:1
julia> status
ERROR: UndefVarError: status not defined
(@v1.7) pkg>
(@v1.7) pkg> status
Status `~/.julia/environments/v1.7/Project.toml`
[c3e4b0f8] Pluto v0.17.2
(@v1.7) pkg>
See what I mean? running status in Pkg only yields Pluto which is correct as I only added Pluto in Julia and PyCall inside Pluto.
Each Pluto notebook typically specifies its own package dependencies, so looking at the default (@v1.7) package environment doesn’t tell you anything about a particular notebook’s environment. Under this paradigm, you need to do all package management within the notebook GUI instead of using Julia’s standard package manager. If you want more fine-grained control of your packages, see this section from the Pluto docs: https://github.com/fonsp/Pluto.jl/wiki/🎁-Package-management#advanced-set-up-an-environment-with-pkgactivate
Hi @stillyslalom YEP that’s what I’m struggling with. I did load PyCall into ONLY the Pluto notebook and it’s working fine ( see my comments above) BUT what I can’t figure out is how to rebuild PyCall inside the Pluto notebook. I had read Fons’s docs ( but thanks for the link) but I couldn’t see how to do the rebuild inside the Pluto gui. Do you know how to do this please?
I don’t want to override Pluto’s Pkg manager as I intend to do all my development inside Pluto. I will NOT be using github, vscode et al.
Within a Pluto cell, do
using Pkg
Pkg.build("PyCall")
The output of this cell will be displayed in the terminal from which you launched Pluto. Once the build process is complete, you’ll want to delete (or comment out) the cell so you don’t re-build PyCall every time you launch the notebook.
Hi @stillyslalom I’m a little worried that approach would turn OFF the new Pluto Pkg manager. Here’s why I am concerned
turning OFF Pluto new Pkg manager
Maybe, and remember I am a noob, this is NOT the case. I love the look and feel of the new approach and don’t want to dig myself into a hole later on by
using Pkg
Pkg.build("PyCall")
What are your thoughts on this?
thanks again
So long as you don’t use Pkg.activate in a Pluto notebook cell, you won’t turn off Pluto’s package management. Please give my instructions a shot.
hi @stillyslalom
so I ran the cells
using Pkg
Pkg.build("PyCall")
and have been watching the terminal in which I started Pluto. It’s been like this for 10 minutes on a 36 core 128gb system so I can’t believe it’s still Building anything.
Did I mess up?
┌ Warning: FYI, running unbundled (dev) frontend
└ @ Pluto ~/.julia/packages/Pluto/7TMtD/src/webserver/WebServer.jl:264
Updating registry at `~/.julia/registries/General.toml`
From worker 2: ┌ Warning: `vendor()` is deprecated, use `BLAS.get_config()` and inspect the output instead
From worker 2: │ caller = npyinitialize() at numpy.jl:67
From worker 2: └ @ PyCall ~/.julia/packages/PyCall/3fwVL/src/numpy.jl:67
From worker 2: Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
From worker 2: Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/4ba3651d33ef76e24fef6a598b63ffd1c5e1cd17/build.log`
the log says
dave@deepthought:~$ cat ~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/4ba3651d33ef76e24fef6a598b63ffd1c5e1cd17/build.log
[ Info: PyCall is using /usr/bin/python3 (Python 3.8.10) at /usr/bin/python3, libpython = /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
[ Info: /home/dave/.julia/packages/PyCall/3fwVL/deps/deps.jl has not changed
[ Info: /home/dave/.julia/prefs/PyCall has not changed
dave@deepthought:~$
running the cell
py_script = pyimport("test_ib_60_sec.py")
still gives the same error.
Python modules are defined by their filename (minus .py), so you should just use pyimport("test_ib_60_sec").
Progress 
@stillyslalom thanks for sticking with me. I changed the script name to test and the cell to
py_script = pyimport("test")
and got
py_script = PyObject <module 'test' from '/home/dave/j_sandbox/j_raw/test.py'>
but no output. It’s supposed to start printing out tickers to stdio but right now nothing on the console. Normally I just run $python3 test.py and it runs the script from a " ‘main’ " and runs an asynch call until it times out ( sleep timer).
I’ll get into that tomorrow BUT thank you so much for sticking with a poor noob and getting me to this point. Most appreciated.