How to run an entire python script in pycall

[anon69491625]

Hi all

I am using PyCall inside Pluto but I can’t run a whole python script, can anyone point me at any examples of this please?

getting to grips with julia using Pluto and it’s fun so far. With a lot of GREAT help from the community I have PyCall able to

py_script = pyimport("test")

BUT every example of PyCall code seems to only allow running segments of the python script. I can’t find any examples that would run a whole script like this one called test.py.

import pandas as pd
import numpy as np
import uuid
env_dir = '/home/dave/j_sandbox/'

dev_env_csv    =  env_dir + '/csv/'  # where to get the expert symbols from


expert_symbols_csv = "tontine_symbols.csv"  # the symbols table with SPX index added AND the category stuff  added 4 9 20


def get_stuff(csv_file):

     df_symbol_to_process_list =  pd.read_csv( csv_file )
              
     return df_symbol_to_process_list
    
if __name__  == '__main__' :

    mac_address = hex(uuid.getnode())
    
    df_expert_symbols = get_stuff (dev_env_csv + expert_symbols_csv)

I am using the latest julia 1.7.0 and the latest Pluto v0.17.2. I am also ONLY using the Pluto package manager ONLY. I am a noob so this is my first real julia program.

[k945]

Have you tried

py"""
your code
"""

Have a look at the usage section in https://github.com/JuliaPy/PyCall.jl

[jling]

if you’re running the whole script and has no interaction between julia, you can simply run it via run() shell interaction

[anon69491625]

hi there
the script “will” interact with my julia code as it would populate a dataframe. I did look into the shell approach but want to try to get this working. Thanks for the suggestion.

[anon69491625]

Hi there

I read it but couldn’t figure out how it was implemented in PyCall. I read it as a separate approach. On your suggestion I went back. Reread it and then found an example

using PyCall
py"""
def print_one_number(my_number):  
    print(my_number)   
    return my_number
"""
my_number_to_print = 10
py"print_one_number"(my_number_to_print)

I still don’t get how this works in running my code I can see how functions are called but not the whole script. My script just runs from main and produces a dataframe. Thanks for the suggestion though

[stevengj]

@anon69491625

BUT every example of PyCall code seems to only allow running segments of the python script. I can’t find any examples that would run a whole script like this one called test.py.

As I understand it, you want to execute an external Python script as if it were in a py"..." macro, not as a separate Python module, so that the globals it defines are available for subsequent py"..." evaluations? i.e. you want the equivalent of execfile("test.py") in Python 2 or exec(open("test.py").read()) in Python 3?

PyCall doesn’t export something like this right now, but it has this facility. Define:

pyinclude(fname) = (PyCall.pyeval_(read(fname, String), PyCall.pynamespace(Main), PyCall.pynamespace(Main), PyCall.Py_file_input, fname); nothing)

and then just do pyinclude("test.py").

[anon69491625]

Hi @stevengj thank you for the reply. You are absolutely correct, I “just” want to run an existing python script (“simple.py”) which is in the same directory as the julia script. I just want to start off on the right foot and it seemed that this was a way to reuse some of my existing code. I have no problem setting it up as a python module if that makes more sense.

As you can see ( sorry about the image I don’t know how to get at the Pluto julia code yet to paste it) I can get at the functions in the python program but not run it. I tried your define and got

UndefVarError: pynamespace not defined

here’s the “code” so far ( again sorry about the image)

[stevengj]

@anon69491625

UndefVarError: pynamespace not defined

There was a typo in my code above — it should have been PyCall.pynamespace (since that function is not exported). I fixed the code above; try it again.

PS. Never post screenshots of code.

[lungben]

@anon69491625

using PyCall
py"""
def print_one_number(my_number):  
    print(my_number)   
    return my_number
"""
my_number_to_print = 10
py"print_one_number"(my_number_to_print)

If you give the Python function a Julia name, you can call it directly from Julia:

using PyCall
py"""
def print_one_number(my_number):  
    print(my_number)   
    return my_number
"""
print_one_number_py = py"print_one_number"  # only the 1-line version of the py macro yields the return value back to Julia!
my_number_to_print = 10
print_one_number_py(my_number_to_print)

[cjdoris]

The following should do the same:

py"""
exec(open("somefile.py").read(), globals(), locals())
"""

[anon69491625]

thank you. I changed the code and there is no error but I can’t see the populated variables. For instance how do I see mac_address which was populated by the “simple.py” script.

VERY sorry about the image, it won’t happen again.

[anon69491625]

hi there
that’s calling a function. I can do that but I want to run the whole script. That’s the question.
thanks though

[stevengj]

@cjdoris

The following should do the same:

Almost. The only problem with doing exec in Python is that it doesn’t know the filename, so that if you have syntax errors in Python it won’t say the filename in the error message.

For that matter, you can also do:

py"""
$$(read("foo.py", String))
"""

I’ll add a @pyinclude("foo.py") macro to a future PyCall release: add pyinclude macro by stevengj · Pull Request #948 · JuliaPy/PyCall.jl · GitHub

[cjdoris]

Ok then

py"""
exec(compile(open("somefile.py").read(), "somefile.py", "exec"), globals(), locals())
"""

[stevengj]

@anon69491625

For instance how do I see mac_address which was populated by the “simple.py” script.

It’s a Python variable, so you need py"mac_address"?

If you are in Pluto, however, evaluating in pynamespace(Main) may be the wrong module — Pluto code may run in some other module? My upcoming @pyinclude macro will fix this because it knows the current module, but in the meantime I would recommend doing:

pyinclude(fname) = py"""
exec(compile(open($fname).read(), $fname, "exec"), globals(), locals())
"""

instead.

[anon69491625]

@cjdoris @stevengj

I tried both but I am certainly missing something here. When I run the cells ( either approach) I would expect to see mac_address and df_expert_symbols in the julia space but I don’t. Running either approach causes no errors, no warnings just nothing.

I you look at the image ( again sorry about that) you’ll see that I can call a function but NOT the script.

what am I missing please?

[stevengj]

@anon69491625

I can’t find any examples that would run a whole script like this one called test.py.

(In general, we tend to avoid code development oriented around “scripts” full of global variables. It’s a lot more flexible to work with local variables and functions, and to translate Python code into Julia line-by-line. That’s perhaps why there hasn’t been much demand for a pyinclude-like function over the years.)

[stevengj]

@anon69491625

I would expect to see mac_address and df_expert_symbols in the julia space

As I wrote above:

@stevengj

It’s a Python variable, so you need py"mac_address" ?

PyCall does not automatically turn Python global variables into Julia global variables; you still have to access the former through py"...".

[anon69491625]

hi @stevengj

a little more confused now, sorry this is all new to me. If I’m running a python script called simple.py what would be the correct Pluto cell to enter?

[lungben]

Just wrap your script into a function on Python side?