# Buffer in running external program in loops

**URL:** <https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087>\
**Category:** Performance\
**Tags:** question, run\
**Created:** [March 18, 2022, 2:02pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087 "2022-03-18T14:02:25Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:02pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/1 "2022-03-18T14:02:25Z")

</div>

I’d like to run system calls _sequentially_ in a for loop, waiting for last call to complete before running the next one.

The _run(command)_ call is nested in a function (_run\_system\_call_ in the example under) which is part of a package that is built with parallel execution in mind (which is fine in 99% of cases) that I’d preferably not modify, so I’d like to avoid the obvious workaround of using _/wait_ in the command to solve the problem on the system side. BTW I’m using Windows, but the problem seems to lie in Julia so I guess it would also be encountered under Linux.

I am using the following code (exemplified):

```julia
for (ic,num) in enumerate(cases)
   run_system_call(cases[ic])
   while *completion criterion not satisfied*
        *update completion criterion*
   end
end

```

That does not work. run\_system\_call is called without error but does not run the system command in practice, obviously leading to deadlock in the while loop. I’ve tried rearranging the for/while loops, it does not help. It seems julia waits for for/while loops (and likely any soft scope) to complete before effectively sending out the system calls, that must be put in a buffer.

Anyone knowing if there is a way to “flush” the system call buffer? Or an idea of workaround?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:12pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/2 "2022-03-18T14:12:42Z")

</div>

I don’t think there is some buffer which isn’t flushed in time.  
You example doesn’t have enough information to have an idea, but have you used the `wait`-parameter of `run`?

```julia
help?> run
search: run trunc truncate round rounding RoundUp RoundDown RoundToZero RoundingMode RoundNearest RoundFromZero

  run(command, args...; wait::Bool = true)

  Run a command object, constructed with backticks (see the Running External Programs section in the manual). Throws
  an error if anything goes wrong, including the process exiting with a non-zero status (when wait is true).

  If wait is false, the process runs asynchronously. You can later wait for it and check its exit status by calling
  success on the returned process object.

  When wait is false, the process' I/O streams are directed to devnull. When wait is true, I/O streams are shared with
  the parent process. Use pipeline to control I/O redirection.

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:13pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/3 "2022-03-18T14:13:47Z")

</div>

> [@ValentinChb](#):
>
> so I’d like to avoid the obvious workaround of using _/wait_ in the command to solve the problem on the system side.

I see…

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:17pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/4 "2022-03-18T14:17:53Z")

</div>

> [@ValentinChb](#):
>
> run\_system\_call is called without error but does not run the system command in practice

The question is, why is the command not executed, or why does the executed command doesn’t create the completion criterion. Perhaps because of redirection of STDOUT and STDERR?

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:17pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/5 "2022-03-18T14:17:55Z")

</div>

Yeah that’s the point. I could solve the problem by tweaking the run\_system\_call function, but I’d rather not modify the package. And anyway, my post is also to raise attention on this weird behaviour of Julia. There must be some kind of buffering, it _is_ waiting for the loop to complete before the system calls take action.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:18pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/6 "2022-03-18T14:18:42Z")

</div>

For the discussion to be in sync, here again my guess:

> [@oheil](#):
>
> Perhaps because of redirection of STDOUT and STDERR?

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:20pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/7 "2022-03-18T14:20:15Z")

</div>

The completion criterion is based on checking whether output files from the system call have been updated. Anyway, the system call should open a new command window for each case so I can see immediately whether the it has been effectively executed or not.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:21pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/8 "2022-03-18T14:21:13Z")

</div>

> [@ValentinChb](#):
>
> system call should open a new command window

Perhaps not. `run` doesn’t open a CMD shell, it directly starts the commands using `exec` calls.

[https://docs.julialang.org/en/v1.7/manual/running-external-programs/](https://docs.julialang.org/en/v1.7/manual/running-external-programs/)

> - The command is never run with a shell. Instead, Julia parses the command syntax directly, appropriately interpolating variables and splitting on words as the shell would, respecting shell quoting syntax. The command is run as `julia` 's immediate child process, using `fork` and `exec` calls.

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:25pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/9 "2022-03-18T14:25:04Z")

</div>

Indeed, sorry for not being thourough here. I’m using `run` to run a batch script that in turns runs `start <process>` in a new window. If I remove the while loop, the code will effectively run the batch script for each case in the for loop, and I’ll be left with multiple instances of that script running in parallel, each in its own window.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:26pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/10 "2022-03-18T14:26:14Z")

</div>

Ok, I try to mimic this here on my windows machine…

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:30pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/11 "2022-03-18T14:30:13Z")

</div>

Thanks a lot, here is some complementary info:  
_run\_system\_call.jl_ should contain something like  
`run(´cmd start script.bat´)`  
and _script.bat_ should contain something like  
`start cmd @cmd /k echo "hello world"`

that should open a new window printing hello world on each. But if it behaves in the same way as on my machine, you won’t see any window popping up.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:34pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/12 "2022-03-18T14:34:34Z")

</div>

> [@ValentinChb](#):
>
> ```julia
> run(`cmd start script.bat`)
> 
> ```

Is this mandatory? because:

```julia
run(`script.bat`)

```

should just be equivalent on windows.  
And this

```julia
start cmd @cmd /k echo "hello world"

```

seems also overly complex. Why not just

```julia
echo "hello world"

```

in the .bat?  
Perhaps these are already artefacts of your search for a solution, so I ask explicitly.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:36pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/13 "2022-03-18T14:36:08Z")

</div>

> [@oheil](#):
>
> `start cmd @cmd /k echo "hello world"`

I see why you do this… ok I will do and set it up here…

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:40pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/14 "2022-03-18T14:40:02Z")

</div>

Works fine so far.  
What is your exact completion criterion?  
When I do this:

```julia
function test()
   run(`test.bat`)
   while ! isfile("test.out")
       sleep(5)
   end
end

test()

```

I get the cmd windows with a prompt (“hello world” at the beginning) and Julia prompt comes back when I say  
`touch "test.out"`  
in this CMD shell.

> [@oheil](#):
>
> ```julia
> run(`test.bat`)
> 
> ```

but doesn’t work with

```julia
run(`cmd start test.bat`)

```

… wait for it

Can you try instead of

```julia
run(`cmd start script.bat`)

```

this:

```julia
run(`cmd /C "start script.bat"`)

```

?

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:47pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/15 "2022-03-18T14:47:19Z")

</div>

Yes I understand your point, in reality that’s quite complex code that is run, the overly complex batch stuff is to get the stdout properly on separate windows in different cases.  
`run(`script.bat`)` will put everything in the julia shell, which I don’t want. That won’t happen if _script.bat_ is in turn running `start cmd /k` (I agree the `@cmd` was not necessary), but that is not always so, depending on cases[ic]. In short there are good reasons to have it like this 🙂

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [March 18, 2022, 2:49pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/16 "2022-03-18T14:49:13Z")

</div>

Can you try instead of

```julia
run(`cmd start script.bat`)

```

this:

```julia
run(`cmd /C "start script.bat"`)

```

?

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:51pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/17 "2022-03-18T14:51:13Z")

</div>

Thanks, sorry for lagging your posts all the time. I’ll try that.

---

<div class="post-metadata">

**Author:** ![lawless-m](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lawless-m/32/30869_2.png) [@lawless-m](https://discourse.julialang.org/u/lawless-m)\
**Post date:** [March 18, 2022, 2:51pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/18 "2022-03-18T14:51:48Z")

</div>

I’m not really helping but can I comment that “system call” already has a specific meaning of calling into the kernel from user space, while what you are doing is running a user space program.

> **[System call](https://en.wikipedia.org/wiki/System_call)**
>
> In computing, a system call (commonly abbreviated to syscall) is the programmatic way in which a computer program requests a service from the operating system\[a\] on which it is executed. This may include hardware-related services (for example, accessing a hard disk drive or accessing the device's camera), creation and execution of new processes, and communication with integral kernel services such as process scheduling. System calls provide an essential interface between a process and the oper...

[https://man7.org/linux/man-pages/man2/syscall.2.html](https://man7.org/linux/man-pages/man2/syscall.2.html)

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 2:55pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/19 "2022-03-18T14:55:22Z")

</div>

Sorry for the terminology 🙂 I guess I’m running an external program by means of syscalls?

---

<div class="post-metadata">

**Author:** ![ValentinChb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valentinchb/32/34668_2.png) [@ValentinChb](https://discourse.julialang.org/u/ValentinChb)\
**Post date:** [March 18, 2022, 3:09pm UTC](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087/20 "2022-03-18T15:09:06Z")

</div>

> [@oheil](#):
>
> Can you try instead of
> 
> ```julia-auto
> run(`cmd start script.bat`)
> 
> ```
> 
> this:
> 
> ```julia-auto
> run(`cmd /C "start script.bat"`)
> 
> ```
> 
> ?

that does not seem to help. Actually I did have the `/c`, but after the `start`. None works. maybe I should be even more precise:  
I used to use

`run(`program.exe inputfile.inp`)`  
(btw sorry for not putting backticks around the command, how to differentiate it from preformatted text in the forum?)

but that put all of stdout in julia shell, while I wanted to have each instance of external program in a separate window to easily monitor its progress. that’s why I’m using

`run(`cmd start /c script.bat`)`

with _script.bat_ running

`start cmd @cmd /k program.exe inputfile.inp`

[Next page](https://discourse.julialang.org/t/buffer-in-running-external-program-in-loops/78087.md?page=2)