# Communication with python and etc
**URL:** https://discourse.julialang.org/t/communication-with-python-and-etc/74767
**Category:** General Usage
**Tags:** question, python, io, scripting
**Created:** [January 17, 2022, 7:43pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767 "2022-01-17T19:43:09Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 17, 2022, 7:43pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/1 "2022-01-17T19:43:09Z")
I have an interactive python script that writes to and asks for input from the terminal multiple times. Is there an easy way to interact with it? I tried to redirect input and output through `Pipe` and `PipeBuffer` but couldn’t figure it out. I also found [this](https://gist.github.com/singularitti/ab3a7ac060e79dda6a88f58d7657e63f) but I can’t extend this for multiple I/O.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 17, 2022, 9:49pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/2 "2022-01-17T21:49:50Z")
Hi @Machine-Teacher,
it would probably helpful if you would post some code excerpt to help us reason about it.
Thanks!
---
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 18, 2022, 7:58am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/3 "2022-01-18T07:58:46Z")
Hi @goerch,
I would like my code to work like this. This is obviously not working code, but perhaps someone can suggest how to implement similar functionality.
```julia
function communicate(cmd::Cmd)
i, o, e = Pipe(), Pipe(), Pipe()
process = run(pipeline(cmd; stdin=i, stdout=o, stderr=e); wait=false)
close(o.in)
close(e.in)
while true
stdout = @async String(read(o))
stderr = @async String(read(e))
println(String(fetch(stdout)))
write(i, readline())
fetch(e) != 0 && break
wait(o)
end
println(fetch(e))
kill(process)
end
```
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 18, 2022, 3:18pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/4 "2022-01-18T15:18:54Z")
Thanks for posting. I’m by no means an expert in this but based on your code I tried something like
```julia
function communicate(cmd::Cmd)
i, o, e = Pipe(), Pipe(), Pipe()
process = run(pipeline(cmd; stdin=i, stdout=o, stderr=e); wait=false)
try
Threads.@threads for stream in [stdin, stdout, stderr]
if stream == stdin
try
while true
s = readline()
println(i, s)
end
catch exception
!(exception isa InterruptException) && rethrow()
end
elseif stream == stdout
while true
s = String(readline(o))
println(s)
end
elseif stream == stderr
while true
s = String(readline(e))
println(s)
end
end
end
catch exception
!(exception isa InterruptException) && rethrow()
finally
kill(process)
end
end
communicate(Cmd(["C:/Users/Win10/AppData/Local/Programs/Julia-1.8.0-DEV/bin/julia.exe"]))
```
Maybe this gets the discussion started. Should this be done with Threads or Tasks?
---
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 19, 2022, 6:34pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/5 "2022-01-19T18:34:28Z")
Thanks for your reply. I believe this would be better done with sequential tasks. After all, the processes (reading, writing) are sequential, I don’t understand why threads should be used for this. In addition, in the general case, the program should continue until the end of the process, but I just can’t implement it.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 19, 2022, 7:13pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/6 "2022-01-19T19:13:31Z")
> [@Machine-Teacher](#):
>
> After all, the processes (reading, writing) are sequential.
This is an assumption isn’t it? And in general we don’t know the exact response of the communication partner (for example how many lines (bytes) of response and when to expect them).
> [@Machine-Teacher](#):
>
> In addition, in the general case, the program should continue until the end of the process.
I believe tear down handling is a special case (I just checked sigint). Which process will end first, parent or child?
---
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 20, 2022, 4:47am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/7 "2022-01-20T04:47:06Z")
> [@goerch](#):
>
> This is an assumption isn’t it? And in general we don’t know the exact response of the communication partner (for example how many lines (bytes) of response and when to expect them).
Yes it is. In general, we don’t know the IO order, but this seems to me a bit more cluttered solution.
> [@goerch](#):
>
> I believe tear down handling is a special case (I just checked sigint). Which process will end first, parent or child?
In the general case, it can be different, but in my case a child process will end first.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 20, 2022, 5:00am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/8 "2022-01-20T05:00:29Z")
I just checked @tkf 's [original](https://tkf.github.io/julia-python-snippets/dev/miscellaneous/#Read-from-and-write-to-subprocess-simultaneously-1) code: this seems to solve the communication problem for one shot execution. I also don’t see how to extend this to fulfill your requirement
> [@Machine-Teacher](#):
>
> I have an interactive python script that writes to and asks for input from the terminal multiple times. Is there an easy way to interact with it?
In absence of other information I believe `Threads` is the way to go.
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [January 20, 2022, 5:15am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/9 "2022-01-20T05:15:39Z")
Using `Threads.@threads for` is not the right tool for this. You’d need to wrap blocking call in a `Threads.@spawn` or `@async`.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 20, 2022, 5:47am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/10 "2022-01-20T05:47:35Z")
> [@tkf](#):
>
> Using `Threads.@threads for` is not the right tool for this. You’d need to wrap blocking call in a `Threads.@spawn` or `@async` .
Thanks! Something like this
```julia
function communicate(cmd::Cmd)
i, o, e = Pipe(), Pipe(), Pipe()
process = run(pipeline(cmd; stdin=i, stdout=o, stderr=e); wait=false)
try
@async (
while true
s = String(readline(o))
println(s)
end
)
@async (
while true
s = String(readline(e))
println(s)
end
)
while true
s = readline()
println(i, s)
end
catch exception
!(exception isa InterruptException) && rethrow()
finally
kill(process)
end
end
communicate(Cmd(["C:/Users/Win10/AppData/Local/Programs/Julia-1.8.0-DEV/bin/julia.exe"]))
```
then?
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [January 20, 2022, 6:40am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/11 "2022-01-20T06:40:12Z")
Yeah, I think you got the idea. But a few more comments:
If you don’t have `@sync` around `@async`, that’s usually a bad idea. The errors in `@async` tasks would be ignored and it’d be hard to debug.
I think you to close the output pipes `o.in` and `e.in` after `run` if you want to stop the tasks when the program exit.
The `while` loops can be written as `for line in eachline(o) ...` and `for line in eachline(stdin) ...`.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 20, 2022, 7:59am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/12 "2022-01-20T07:59:56Z")
Thanks again. I tried to incorporate your suggestions:
```julia
function communicate(cmd::Cmd)
i, o, e = Pipe(), Pipe(), Pipe()
process = run(pipeline(cmd; stdin=i, stdout=o, stderr=e); wait=false)
try
@sync for s in eachline()
@async (
for s in eachline(o)
println(s)
end
)
@async (
for s in eachline(e)
println(s)
end
)
println(i, s)
end
catch exception
!(exception isa InterruptException) && rethrow()
finally
kill(process)
close(o.in)
close(e.in)
end
end
communicate(Cmd(["C:/Users/Win10/AppData/Local/Programs/Julia-1.8.0-DEV/bin/julia.exe"]))
```
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [January 21, 2022, 6:02am UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/13 "2022-01-21T06:02:28Z")
You don’t want to schedule tasks for each line. Also, the input endpoint should be closed before waiting for the “reader” tasks (that automatically happens at the end of the `@sync` block). I think you’d need something like this (untested):
```julia
function communicate(cmd::Cmd)
i, o, e = Pipe(), Pipe(), Pipe()
process = run(pipeline(cmd; stdin=i, stdout=o, stderr=e); wait=false)
close(o.in)
close(e.in)
@sync try
@async for s in eachline(o)
println(s)
end
@async for s in eachline(e)
println(s)
end
for s in eachline()
println(i, s)
end
catch exception
!(exception isa InterruptException) && rethrow()
finally
kill(process)
end
end
```
---
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 22, 2022, 12:46pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/14 "2022-01-22T12:46:56Z")
Thanks, it is more closer. I tested your solution and after the process ends it gives all the output in one go. I would like to get the current output while the script is still running. And is it really impossible to end the script without directly interrupting it?
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 22, 2022, 1:45pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/15 "2022-01-22T13:45:30Z")
> [@Machine-Teacher](#):
>
> I have an interactive python script that writes to and asks for input from the terminal multiple times.
Refactor your script so that you can call it as a library, by calling functions (via PyCall in Julia) rather than I/O. Don’t write scripts for code that you want to re-use, write functions/modules/classes.
---
### Author:  [@Machine-Teacher](https://discourse.julialang.org/u/Machine-Teacher)
#### Post date: [January 22, 2022, 3:07pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/16 "2022-01-22T15:07:59Z")
> [@stevengj](#):
>
> > [@Machine-Teacher](#):
> >
> > I have an interactive python script that writes to and asks for input from the terminal multiple times.
>
> Refactor your script so that you can call it as a library, by calling functions (via PyCall in Julia) rather than I/O. Don’t write scripts for code that you want to re-use, write functions/modules/classes.
Is it really that bad?
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 22, 2022, 3:10pm UTC](https://discourse.julialang.org/t/communication-with-python-and-etc/74767/17 "2022-01-22T15:10:54Z")
> [@Machine-Teacher](#):
>
> Is it really that bad?
Yes. (This is regardless of what language you are using.)