Importing local python code directory function in Julia

raman_kumar · June 27, 2024, 3:43pm

Please help me in importing local python code directory within Julia. I am analyzing astrophysical simulation data output from Atheena++. I am writing code in julia to plot data manipulated by python script. My code is : but i am still getting error.

using  Interpolations, TensorOperations, DelimitedFiles, PythonCall
athinput=run("/home/raman/Pictures/MikiFor2to3/pyfiles/read_athinput.py")
runname='/'
inputfile=runname+'home/raman/Pictures/sane00.athinput'
datalist ="./home/raman/Pictures/sane00.athinput/"

I am getting error

   ~  julia Fluxes.jl                                                  ✔ 
ERROR: LoadError: MethodError: no method matching run(::String)

Closest candidates are:
  run(::Base.AbstractCmd, Any...; wait)
   @ Base process.jl:477

Stacktrace:
 [1] top-level scope
   @ ~/Fluxes.jl:2
in expression starting at /home/raman/Fluxes.jl:2

abraemer · June 28, 2024, 9:31am

Your code is a mess.

The error you see MethodError: no method matching run(::String) tells you that you are using run wrong. The correct way to to use it is with: run(`your-cmd-here`) note the different quotation marks!
You are not using the output of run that you store in athinput. But run likely does not return what you expect it to anyways and I can’t really guess what you expect it to return since you didn’t provide any explanation or even the Python script. Perhaps you’ll want to readchomp(`your-cmd-here`) instead of run
This line inputfile=runname+'home/raman/Pictures/sane00.athinput' this pure Python code that snuck into your Julia script. In Julia 'a' denotes a character not a string and thus 'more-than-one-char' throws. Also string concatenation is * in Julia.
Later in this script you use the file variable which you did not define. Perhaps it should be replaced with athinput?

Please put at least some effort into your posts here! People helping you here do this on their own free time. So very low effort post like yours might just be ignored.

This certainly was the last answer you’ll get from me, if your post does not show any sign that you at least tried to solve your problem i.e. by reading the error message.

raman_kumar · June 28, 2024, 10:38am

How to import local .py python file from your laptop in Julia? I am unable to understand this link. How can we import python files? · JuliaPy/PythonCall.jl · Discussion #145 · GitHub

   ~  julia Fluxes.jl                                                  ✔ 
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Resolving changes
             + pyexec
    CondaPkg Installing packages
             │ /home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
             │ -r /home/raman/micromamba
             │ install
             │ -y
             │ -p /home/raman/.julia/environments/v1.10/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0']
             │ pyexec[version='*']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge
conda-forge/linux-64                                          No change
conda-forge/noarch                                  15.2MB @ 927.2kB/s 16.5s
error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ pyexec *  does not exist (perhaps a typo or a missing channel).
critical libmamba Could not solve for environment specs
ERROR: LoadError: InitError: failed process: Process(`/home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba -r /home/raman/micromamba install -y -p /home/raman/.julia/environments/v1.10/.CondaPkg/env --override-channels --no-channel-priority "libstdcxx-ng[version='>=3.4,<13.0']" "pyexec[version='*']" "python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']" -c conda-forge`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:398
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:299
  [6] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:287
  [7] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:525
  [8] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:405
  [9] envdir()
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/env.jl:70
 [10] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/S5MOg/src/C/context.jl:63
 [11] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/S5MOg/src/C/C.jl:23
 [12] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
 [13] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
 [14] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1067
 [15] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1581
 [16] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1938
 [17] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [18] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [19] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [20] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [21] macro expansion
    @ ./loading.jl:1790 [inlined]
 [22] macro expansion
    @ ./lock.jl:267 [inlined]
 [23] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [24] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [25] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [26] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
during initialization of module C
in expression starting at /home/raman/Fluxes.jl:1

ufechner7 · June 28, 2024, 12:36pm

You cannot import Python files, only Python packages. Can you explain what you actually want to do? What is your goal? Execute Python scripts? Call Python functions? Instantiate Python classes? Or something else?

raman_kumar · June 28, 2024, 2:57pm

Yes, I want to execute python scripts within julia. I have simulation data with will be manipulated by lengthy python scripts

    ~/Pi/MikiFor2to3/pyfiles  ls                                        ✔ 
athena_read.py   fm_torus.pyc  metric.py    read_athinput.py   read_data.pyc
athena_read.pyc  __init__.py   metric.pyc   read_athinput.pyc  unimesh.py
fm_torus.py      __init__.pyc  __pycache__  read_data.py       unimesh.pyc

and then i will feed output to my julia scripts which will plot finally.
I would like to convert all those python scripts in julia but it will take a lot of time, so i just want to use julia only at last stage of data analysis.

ufechner7 · June 28, 2024, 3:21pm

If you want to run the whole script and it has no direct interaction with Julia, you can simply run it via run() shell interaction. See: Running External Programs · The Julia Language

Example:

julia> run(`python hello.py`)
Hello
Process(`python hello.py`, ProcessExited(0))

assuming a file hello.py exists in the current directory and Python is installed and in the search path.

mkitti · June 28, 2024, 3:49pm

See the documentation of CondaPkg.jl:

julia> using CondaPkg
julia> # now press ] to enter the Pkg REPL
pkg> conda status                # see what we have installed
pkg> conda add python perl       # adds conda packages
pkg> conda pip_add build         # adds pip packages
pkg> conda rm perl               # removes conda packages
pkg> conda run python --version  # runs the given command in the conda environment
pkg> conda update                # update conda and pip installed packages

In particular note the conda pip_add build command to help you load Python packages installed via pip.

In particular, see

raman_kumar · June 29, 2024, 2:56am

I want to import read_data() function
dta=read_data(datafile,athinput)
from python script to julia script. This python function takes input data from other python scripts output.

mkitti · June 29, 2024, 3:11am

Yes, I know. First you have to make sure you have the Python packages setup. In your case pyfiles is not on conda-forge so you must use pip via CondaPkg.jl as I pointed out above.

Then you might want you to load your Python file by using pyexec to include the Python file.

raman_kumar · June 29, 2024, 3:12am

@mkitti I have modified pyfiles in my local directory. I don’t need to add official pyfiles as they would not work for me as i have to do some modifications. My local modified pyfiles is having extra .py files

   ~/Pi/MikiFor2to3/pyfiles  ls                                        ✔ 
athena_read.py   fm_torus.pyc  metric.py    read_athinput.py   read_data.pyc
athena_read.pyc  __init__.py   metric.pyc   read_athinput.pyc  unimesh.py
fm_torus.py      __init__.pyc  __pycache__  read_data.py       unimesh.pyc

Benny · June 29, 2024, 5:27am

I’m sorry to repeat this, but this is still a Python problem. You did not learn how to add your local package to an environment (conda being more relevant here) as suggested in the previous thread. Your athinput=run("...") line doesn’t even begin to resemble Python’s import statement or PythonCall’s pyimport wrapper, and it makes me suspect that you were just running .py files instead of importing packages at all. pyexec wraps Python’s exec, but I heavily recommend you don’t do such an unidiomatic practice to submodules of pyfiles, assuming you intended it to have a package structure. Please learn more about Python in the appropriate Python forums where you can expect many more knowledgeable Python users.

mkitti · June 29, 2024, 7:05am

using Pkg
Pkg.activate(".")
Pkg.add(["PythonCall", "CondaPkg"])
# maybe skip the next line?
run(`git clone https://github.com/tianjos/pyfiles`)
using CondaPkg
CondaPkg.add_pip("pyfiles", version="@./pyfiles")
using PythonCall
const pyfiles = PythonCall.pyimport("pyfiles")

ufechner7 · June 29, 2024, 7:28am

UPDATE:
The code above might already work if the folder with your version of pyfiles is a subfolder of your Julia project.

raman_kumar · June 30, 2024, 10:56am

I have python package pyfiles in local directory in my laptop at location: /home/raman/Pictures/MikiFor2to3/pyfiles/ How can i import functions in it from Julia?

mkitti · June 30, 2024, 10:59am

See GitHub - JuliaPy/CondaPkg.jl: Add Conda dependencies to your Julia project .

raman_kumar · June 30, 2024, 11:05am

I am getting error

    ~  julia Fluxes.jl                                                  ✔ 
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Resolving changes
             + build (pip)
             + pip
             + pyexec
    CondaPkg Installing packages
             │ /home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
             │ -r /home/raman/micromamba
             │ install
             │ -y
             │ -p /home/raman/.julia/environments/v1.10/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0']
             │ pip[version='>=22.0.0']
             │ pyexec[version='*']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ pyexec *  does not exist (perhaps a typo or a missing channel).
critical libmamba Could not solve for environment specs
ERROR: LoadError: InitError: failed process: Process(`/home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba -r /home/raman/micromamba install -y -p /home/raman/.julia/environments/v1.10/.CondaPkg/env --override-channels --no-channel-priority "libstdcxx-ng[version='>=3.4,<13.0']" "pip[version='>=22.0.0']" "pyexec[version='*']" "python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']" -c conda-forge`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:398
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:299
  [6] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:287
  [7] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:525
  [8] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:405
  [9] envdir()
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/env.jl:70
 [10] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/S5MOg/src/C/context.jl:63
 [11] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/S5MOg/src/C/C.jl:23
 [12] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
 [13] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
 [14] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1067
 [15] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1581
 [16] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1938
 [17] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [18] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [19] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [20] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [21] macro expansion
    @ ./loading.jl:1790 [inlined]
 [22] macro expansion
    @ ./lock.jl:267 [inlined]
 [23] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [24] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [25] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [26] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
during initialization of module C
in expression starting at /home/raman/Fluxes.jl:1

when i run Fluxes.jl script having lines:

using   Pkg, Interpolations, TensorOperations, DelimitedFiles, PythonCall, CondaPkg
Pkg.activate(".")
CondaPkg.add_pip("pyfiles", version="@./home/raman/Pictures/MikiFor2to3/pyfiles/")
athinput=run(`/home/raman/Pictures/MikiFor2to3/pyfiles/read_athinput.py`))
datafile ="./home/raman/Pictures/sane00.athinput/"

and your post Importing local python code directory function in Julia - #9 by mkitti
gives

ERROR

https://discourse.julialang.org/t/importing-local-python-code-directory-function-in-julia/116315/9?u=raman_kumar

mkitti · June 30, 2024, 11:24am

You probably need to edit your CondaPkg.toml or figure out how to remove pyexec or whatever that is.

raman_kumar · June 30, 2024, 11:36am

Thanks, I removed pyexec by using conda rm pyexec.
But now on running code

using   Pkg, Interpolations, TensorOperations, DelimitedFiles, PythonCall, CondaPkg
Pkg.activate(".")
CondaPkg.add_pip("pyfiles", version="@./home/raman/Pictures/MikiFor2to3/pyfiles/")
athinput=run(`/home/raman/Pictures/MikiFor2to3/pyfiles/read_athinput.py`))
datafile ="./home/raman/Pictures/sane00.athinput/"

i am getting

Error

   ~  julia Fluxes.jl                                        1 ✘  16s  
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Dependencies already up to date
  Activating new project at `~`
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Dependencies already up to date
ERROR: LoadError: IOError: could not spawn `/home/raman/Pictures/MikiFor2to3/pyfiles/read_athinput.py`: permission denied (EACCES)
Stacktrace:
 [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}})
   @ Base ./process.jl:128
 [2] #784
   @ ./process.jl:139 [inlined]
 [3] setup_stdios(f::Base.var"#784#785"{Cmd}, stdios::Vector{Union{RawFD, IO}})
   @ Base ./process.jl:223
 [4] _spawn
   @ ./process.jl:138 [inlined]
 [5] run(::Cmd; wait::Bool)
   @ Base ./process.jl:479
 [6] run(::Cmd)
   @ Base ./process.jl:477
 [7] top-level scope
   @ ~/Fluxes.jl:4
in expression starting at /home/raman/Fluxes.jl:4

mkitti · June 30, 2024, 12:38pm

Here you are trying to execute a Python script as another process. I do not think that is your intention.

raman_kumar · June 30, 2024, 12:39pm

What should i do to import python script read_data() function in Julia?