# How to run an entire python script in pycall

**URL:** <https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414>\
**Category:** New to Julia\
**Tags:** pycall\
**Created:** [December 21, 2021, 3:37am UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414 "2021-12-21T03:37:59Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 21, 2021, 1:41pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/21 "2021-12-21T13:41:31Z")

</div>

Hi there @stevengj  
I understand completely, I’ll go back and revisit my python scripts to see what I can do to be more structured in my approach. I don’t want to start off on the wrong foot.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 21, 2021, 1:43pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/22 "2021-12-21T13:43:24Z")

</div>

@lungben  
certainly worth a try and thanks for the suggestion. @stevengj pointed out that the real approach should be to call the functions not the script. I’ll go back and revisit my python scripts to see what I can do to make it more modular.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 21, 2021, 1:55pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/23 "2021-12-21T13:55:29Z")

</div>

tried that but didn’t work. I think I need to go back and REALLY look into the aims and aspirations of PyCall. I suspect that this is a massive misread on my part. I “thought” calling the python script would populate the julia space with the python variables ( dataframes, variables) and it does PERFECTLY if you call the function. I understand, NOW, that calling the whole script would be a poor idea and I’ll go back and revisit my approach.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 21, 2021, 9:37pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/24 "2021-12-21T21:37:02Z")

</div>

to all that tried to help, thank you all so much. I read through the splendid PyCall docs and pondered my options. In the end I think it’s a better approach for me to leave the existing python scripts mostly unchanged and consider an interprocess communication architecture uni directional from python to julia. The python scripts login to a financial data feed and gather real time data, normally I would do all the data manipulation in python ( numpy, pandas et al) and use dash plotly to represent the results. It wouldn’t take too much for the python scripts to “write” the datastream to some form of IPC network ( I use Linux) and have the julia script read it.  
What I used in PyCall was excellent and I can see it’s value but in my use case I think that the IPC route would be a better option.  
thank you all for trying to help a poor noob, I certainly learnt some things.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 12:15pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/25 "2021-12-22T12:15:38Z")

</div>

hi @lungben  
I don’t like giving up on things, especially when I think my adventures could help others. That said I would like to investigate your concept of wrapping the code below in a function and calling the whole thing from PyCall ( awesome package by the way). Are there any gotcha’s that you can see in this approach?

The “problem” with julia ( like linux or python) is that there are so many possible approaches and history dictates I will take ALL of the poor ones before stumbling on the right one. My usual approach is to just jump in, make mistakes and then spend too much time duct taping when I should just take time to start off on the right foot. Any tips you might have would be appreciated.

```julia
import pandas as pd
import numpy as np
import uuid
env_dir = '/home/dave/j_sandbox/'

dev_env_csv = env_dir + '/csv/' # where to get the expert symbols from

expert_symbols_csv = "tontine_symbols.csv" # the symbols table with SPX index added AND the category stuff added 4 9 20

def get_stuff(csv_file):

     df_symbol_to_process_list = pd.read_csv( csv_file )
              
     return df_symbol_to_process_list
    
if __name__ == ' __main__' :

    mac_address = hex(uuid.getnode())
    
    df_expert_symbols = get_stuff (dev_env_csv + expert_symbols_csv)

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 12:23pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/26 "2021-12-22T12:23:30Z")

</div>

> [@anon69491625](#):
>
> In the end I think it’s a better approach for me to leave the existing python scripts mostly unchanged and consider an interprocess communication architecture uni directional from python to julia.

This seems way more complicated to me. Why not just run the script as-is (note: the latest PyCall release has a `@pyinclude("test.py")` macro) and just fetch the global(s) you want with `py"someglobal"`?

The advice about not using scripts full of globals (writing functions and other structured-programming constructs instead), is just general software-engineering advice; it’s not necessary for calling Python from Julia.

PS. Note that I don’t think ` __name__ == ' __main__'` will be `True` in Python if you execute a file with `exec` as we are doing here, so you could just replace that with `if True:`

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 12:41pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/27 "2021-12-22T12:41:24Z")

</div>

Hi @stevengj

thanks for sticking with me on this. I think I mentioned that if there was a complicated and boneheaded approach, that would be my first port of call 🙂 so instead of buckling down and REALLY considering what everyone has been saying I concentrated ( last night) on my goto approach of getting the python scripts to generate a stream of text messages which I planned on writing a finite state machine to consume. You see what you are dealing with here!

I sat back this morning and considered @lungben and your observations ( thanks for not laughing by the way) and put aside my whiteboarding to revisit PyCall ( outstanding package by the way).

For some reason my attempt to run the package as is yielded no results as I couldn’t access

```julia
df_expert_symbols
mac_address

```

In the Pluto cell after I had run the python script. That threw me off. Right now I can’t post the code as I don’t know how to do that but I’ll figure it out. Though it’s in the image ( sorry) I sent earlier on.

AHA you see you are giving me an out. the lure of globals beckons and I have to resist! At any time I am one step away from a GOTO, tempt me not 🙂

To be honest I would like to just dump the ` __name__ == ' __main__'` as it serves no purpose here. It’s only there just in case I wanted to send args to the cronjob that runs the python script. Thanks for pointing that out.

I am determined to get this working with PyCall given the more I sit down and read about it the more impressed I am.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 2:06pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/28 "2021-12-22T14:06:40Z")

</div>

> [@anon69491625](#):
>
> my attempt to run the package as is yielded no results as I couldn’t access
> 
> ```julia
> df_expert_symbols
> mac_address
> 
> ```
> 
> In the Pluto cell after I had run the python script

As I’ve said repeatedly, these are not _Julia_ globals, they are _Python_ globals. You would access them with `py"df_expert_symbols"` and `py"mac_address"` in Julia (after ditching the ` __name__ == ' __main__'`). (Also, update to the latest PyCall and use `@pyinclude("test.py")`.)

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 2:10pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/29 "2021-12-22T14:10:52Z")

</div>

Hi @stevengj and I did try `py"df_expert_symbols` BUT that was before your observation that the `--name__` functionality needs to go. Sorry for the confusion. When I get back to the machine I’ll make the changes and see what happens. Thanks again for sticking with this.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 2:58pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/30 "2021-12-22T14:58:53Z")

</div>

> [@stevengj](#):
>
> PS. Note that I don’t think ` __name__ == ' __main__'` will be `True` in Python if you execute a file with `exec` as we are doing here, so you could just replace that with `if True:`

Actually, it ` __name__ == ' __main__'` is true by default, so it should be fine in PyCall:

```julia
julia> py" __name__"
" __main__"

julia> py" __name__ == ' __main__'"
true

```

But, as I said above, if you are in Pluto, you should be sure to update to the latest version of PyCall and use the built-in `@pyinclude("test.py")` macro rather than the original `pyinclude` function I posted, since the latter may use the wrong module’s globals for Pluto.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 3:02pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/31 "2021-12-22T15:02:11Z")

</div>

hi there  
will do on all counts. Trying to dump all globals, Thanks for the advice and guidance. wish me luck 😉

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 3:19pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/32 "2021-12-22T15:19:08Z")

</div>

By the way, there is also a problem with doing this kind of thing in Pluto, because Pluto doesn’t know about the dependencies implied by using Python globals — it doesn’t know that a Python global depends on running your Python script, so it may execute the cells in the wrong order unless you run the Python script in the same cell as where you try to fetch the global.

(You could alternatively run it in the REPL or in a Jupyter notebook.)

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 4:51pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/33 "2021-12-22T16:51:42Z")

</div>

noted and thanks. I AM using Pluto exclusively for my development so any gotcha’s are welcome so I can watch out for them. My understanding of Pluto is that it uses in order cell dependencies so I’d expect it to always execute the cells in the order they appear on the gui. I’ll watch out for out of order completion though!.

avoiding the REPL not because it isn’t great I just like simple things, for example in coding python I have always used Idle3. I don’t really want to use vstudio or Atom.  
ALSO I am going to investigate calling Pluto NB from the webapi functionality so I’ll have a julia script running cells and using the features of Pluto to alter the cell’'s behavior.

thanks again for all your help

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 4:56pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/34 "2021-12-22T16:56:37Z")

</div>

> [@anon69491625](#):
>
> My understanding of Pluto is that it uses in order cell dependencies so I’d expect it to always execute the cells in the order they appear on the gui.

Nope, it analyzes the code to figure out dependencies, so that if you have a cell `x = 3` and then change it to `x = 4`, it will re-run all cells that depend on `x` even if they are listed _previously_ in the GUI) See the [Pluto FAQ](https://github.com/fonsp/Pluto.jl/wiki/%F0%9F%92%A1-Choosing-Pluto.jl):

> Pluto.jl figures out the _dependency graph_ between cells, and knows exactly which cells to re-evaluate, in which order. A cell never executes twice.

However, Pluto can only analyze dependencies in _Julia_ code, and has no way of inferring the correct dependencies for _Python_ code that you execute via `py"..."` or `@pyinclude` AFAIK.

So the order in which Pluto executes code that has hidden Python dependencies might be unreliable. In contrast, Jupyter notebooks execute cells only when you run them explicitly, so you manually control the order of execution.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 22, 2021, 5:31pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/35 "2021-12-22T17:31:40Z")

</div>

when I said cells I meant Pluto. It makes sense that Pluto can only build the dependency graph in julia world and not python. I figured as much but stated it poorly. Thanks for the heads up. I will be VERY careful. I’m hoping that this will keep me under control and avoid GOTO mindset 🙂

I’m not going to be using Jupyter as I have settled on Pluto.

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [December 22, 2021, 9:02pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/36 "2021-12-22T21:02:04Z")

</div>

When you always give Julia names to Python objects, Pluto should get the dependencies correctly.  
Example:

```julia
my_func(a, 9) # my_func and a are defined later, but Pluto should figure this out

begin
py"""
def my_func(a, b):
    return a*b
"""
my_func = py"my_func" # give the Python function a Julia name
end

begin
py"""
a = 42*7
"""
a = py"a"
end

```

Edit: as @stevengj pointed out, the assignment to the Julia name must be in the same Pluto cell as its definition on Python side.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 22, 2021, 9:14pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/37 "2021-12-22T21:14:53Z")

</div>

> [@lungben](#):
>
> When you always give Julia names to Python objects, Pluto should get the dependencies correctly.

(_If_ you do so in the _same cell_ where the Python objects are defined, e.g. in the same cell that calls `py"def ..."` or `@pyinclude`.)

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [December 22, 2021, 9:16pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/38 "2021-12-22T21:16:18Z")

</div>

Right, I missed to point that out, fading memories, thanks!

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 23, 2021, 2:15am UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/39 "2021-12-23T02:15:45Z")

</div>

thank you for the example really helps.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [December 23, 2021, 12:33pm UTC](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414/40 "2021-12-23T12:33:17Z")

</div>

Hi all  
not given up on this, just a sync and stabilize period. The more I read about julia the more there is to like. I don’t want to mess this one up so I’m spending a few days reading books and watching videos like this one  
[developing julia packages](https://www.youtube.com/watch?v=QVmU29rCjaA) which is excellent. I want to make sure I start off on the right path. Thank you all for the guidance. The PyCall package is wonderful and I want to make sure I don’t mess up. When I get this working I’ll post my “code” so you can all have a good laugh 🙂

have a happy christmas and a safe new year.

[Previous page](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414.md?page=1)

[Next page](https://discourse.julialang.org/t/how-to-run-an-entire-python-script-in-pycall/73414.md?page=3)