Right way to run external program in parallel?

[Tetrakai]

I could not figure this out all day, until finally this answer saved me:

run(`bash -c """cd temp_dir && some_cmd"""`)

how to change CWD for a command executed in Julia?

Is there a better way to do this?

In particular I want to run a program designed to log results sequentially each time it is run. Ie by writing/appending to files in its working directory. Currently I duplicate the directory, run separate instances, then combine the results. It works but this seems very hacky.

[StevenSiew]

Have you considered writing a shell script?

1. run external program
2. after external program finish, then run julia program

Is there a reason to run both julia program and external program in parallel?

[nhz2]

I use

run(Cmd(`some_cmd some_input`; dir=temp_dir))

inside an asyncmap to run multiple instances of a program in parallel.

[Tetrakai]

Its essentially the innerloop of my simulation. You can get some idea from my earlier question here:

What is the right way to store/record the results of this nested simulation?

I wrote this after seeing the responses to my last question. From the responses I can tell my problem was not understood. What this simulation does is play a rudimentary “game” between pairs of two teams once per “week” and record the player results and league standings. Since the outcome is stochastic this is repeated multiple times. In actuality, the ratings/parameters of the players would change each simulation step and the output would be compared to other stats, but I left that out for no…

Essentially I am running a stochastic game engine in parallel (ie, simultaneous games each week, multiple replications of each season), comparing the results to a baseline, modifying the skill ratings of the players… repeat until the combination of ratings produce “realistic” results.

But I think you are right that perhaps I can use GNU parallel for part of this. Eg: File I/O in gnu parallel - Stack Overflow