Buffer in running external program in loops

General Usage Performance
1

I’d like to run system calls sequentially in a for loop, waiting for last call to complete before running the next one.

The run(command) call is nested in a function (run_system_call in the example under) which is part of a package that is built with parallel execution in mind (which is fine in 99% of cases) that I’d preferably not modify, so I’d like to avoid the obvious workaround of using /wait in the command to solve the problem on the system side. BTW I’m using Windows, but the problem seems to lie in Julia so I guess it would also be encountered under Linux.

I am using the following code (exemplified):

for (ic,num) in enumerate(cases)
   run_system_call(cases[ic])
   while *completion criterion not satisfied*
        *update completion criterion*
   end
end

That does not work. run_system_call is called without error but does not run the system command in practice, obviously leading to deadlock in the while loop. I’ve tried rearranging the for/while loops, it does not help. It seems julia waits for for/while loops (and likely any soft scope) to complete before effectively sending out the system calls, that must be put in a buffer.

Anyone knowing if there is a way to “flush” the system call buffer? Or an idea of workaround?

2

I don’t think there is some buffer which isn’t flushed in time.
You example doesn’t have enough information to have an idea, but have you used the wait-parameter of run?

help?> run
search: run trunc truncate round rounding RoundUp RoundDown RoundToZero RoundingMode RoundNearest RoundFromZero

  run(command, args...; wait::Bool = true)

  Run a command object, constructed with backticks (see the Running External Programs section in the manual). Throws
  an error if anything goes wrong, including the process exiting with a non-zero status (when wait is true).

  If wait is false, the process runs asynchronously. You can later wait for it and check its exit status by calling
  success on the returned process object.

  When wait is false, the process' I/O streams are directed to devnull. When wait is true, I/O streams are shared with
  the parent process. Use pipeline to control I/O redirection.
3

I see…

4

The question is, why is the command not executed, or why does the executed command doesn’t create the completion criterion. Perhaps because of redirection of STDOUT and STDERR?

5

Yeah that’s the point. I could solve the problem by tweaking the run_system_call function, but I’d rather not modify the package. And anyway, my post is also to raise attention on this weird behaviour of Julia. There must be some kind of buffering, it is waiting for the loop to complete before the system calls take action.

6

For the discussion to be in sync, here again my guess:

7

The completion criterion is based on checking whether output files from the system call have been updated. Anyway, the system call should open a new command window for each case so I can see immediately whether the it has been effectively executed or not.

8

Perhaps not. run doesn’t open a CMD shell, it directly starts the commands using exec calls.

https://docs.julialang.org/en/v1.7/manual/running-external-programs/

  • The command is never run with a shell. Instead, Julia parses the command syntax directly, appropriately interpolating variables and splitting on words as the shell would, respecting shell quoting syntax. The command is run as julia 's immediate child process, using fork and exec calls.
9

Indeed, sorry for not being thourough here. I’m using run to run a batch script that in turns runs start <process> in a new window. If I remove the while loop, the code will effectively run the batch script for each case in the for loop, and I’ll be left with multiple instances of that script running in parallel, each in its own window.

10

Ok, I try to mimic this here on my windows machine…

11

Thanks a lot, here is some complementary info:
run_system_call.jl should contain something like
run(´cmd start script.bat´)
and script.bat should contain something like
start cmd @cmd /k echo "hello world"

that should open a new window printing hello world on each. But if it behaves in the same way as on my machine, you won’t see any window popping up.

12

Is this mandatory? because:

run(`script.bat`)

should just be equivalent on windows.
And this

start cmd @cmd /k echo "hello world"

seems also overly complex. Why not just

echo "hello world"

in the .bat?
Perhaps these are already artefacts of your search for a solution, so I ask explicitly.

13

I see why you do this… ok I will do and set it up here…

14

Works fine so far.
What is your exact completion criterion?
When I do this:

function test()
   run(`test.bat`)
   while ! isfile("test.out")
       sleep(5)
   end
end

test()

I get the cmd windows with a prompt (“hello world” at the beginning) and Julia prompt comes back when I say
touch "test.out"
in this CMD shell.

but doesn’t work with

run(`cmd start test.bat`)

… wait for it

Can you try instead of

run(`cmd start script.bat`)

this:

run(`cmd /C "start script.bat"`)

?

1 Like
15

Yes I understand your point, in reality that’s quite complex code that is run, the overly complex batch stuff is to get the stdout properly on separate windows in different cases.
run(script.bat) will put everything in the julia shell, which I don’t want. That won’t happen if script.bat is in turn running start cmd /k (I agree the @cmd was not necessary), but that is not always so, depending on cases[ic]. In short there are good reasons to have it like this :slight_smile:

16

Can you try instead of

run(`cmd start script.bat`)

this:

run(`cmd /C "start script.bat"`)

?

17

Thanks, sorry for lagging your posts all the time. I’ll try that.

18

I’m not really helping but can I comment that “system call” already has a specific meaning of calling into the kernel from user space, while what you are doing is running a user space program.

https://man7.org/linux/man-pages/man2/syscall.2.html

1 Like
19

Sorry for the terminology :slight_smile: I guess I’m running an external program by means of syscalls?

20

that does not seem to help. Actually I did have the /c, but after the start. None works. maybe I should be even more precise:
I used to use

run(program.exe inputfile.inp)
(btw sorry for not putting backticks around the command, how to differentiate it from preformatted text in the forum?)

but that put all of stdout in julia shell, while I wanted to have each instance of external program in a separate window to easily monitor its progress. that’s why I’m using

run(cmd start /c script.bat)

with script.bat running

start cmd @cmd /k program.exe inputfile.inp