# How to launch several run cmd in parallel
**URL:** https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862
**Category:** General Usage
**Tags:** question
**Created:** [September 28, 2021, 9:46am UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862 "2021-09-28T09:46:31Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@RCHG](https://discourse.julialang.org/u/RCHG)
#### Post date: [September 28, 2021, 9:46am UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862/1 "2021-09-28T09:46:31Z")
Hello,
My question is probably easy to solve, but I got confused by the several packages and proposals (@sync, @async, Distributed etc…). In the most simple case there is a script in Julia. This script has a loop that run bash script that run a compiled code (several times in a serial loop).
```julia
for param in [a1,a2,a3,......,a100]
basedir = Base.Filesystem.pwd()
Base.Filesystem.cd(param)
run(`bash loop.sh`) # this launch in serial several programs.
Base.Filesystem.cd(basedir)
end
```
I would like to launch these bash scripts in parallel in such way that I am using all time k cores but not more. So it will launch initially k processes, and when one of the cores is free it run another of these bash loop.sh until the for loop in param is completed.
Thank you very much
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [September 29, 2021, 5:47am UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862/2 "2021-09-29T05:47:15Z")
Maybe you can use a `Semaphore` of size equal to the number of cores, and then call
```julia
sem = Base.Semaphore(8) # if you have 8 cores
for loop
@async begin
Base.acquire(sem)
run(`bash loop.sh`)
Base.release(sem)
end
end
```
You’d have to be careful with the `Base.Filesystem.cd(param)` stuff though, I guess that’s a global state?
---
### Author:  [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)
#### Post date: [September 29, 2021, 11:48am UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862/3 "2021-09-29T11:48:39Z")
If you are on Linux you could also run from Julia [GNU Parallel](https://www.gnu.org/software/parallel/) that does exactly that: it takes a text file with commands (I use a single command per line, I don’t know if you can use it in other ways) and run the first k-cores commands and then as soon as one ends it continues with the remaining commands…
For example in my model I have a “runscenarios.sh” file that contains:
```julia
#!/bin/bash
./run_single_scenario.sh 'scenarioName1'
./run_single_scenario.sh 'scenarioName2'
./run_single_scenario.sh 'inputFile1' 'scenarioName3'
./run_single_scenario.sh 'inputFile2' 'scenarioName4'
...
```
and then I run it with: `parallel --jobs -a runscenarios.sh `
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [September 29, 2021, 10:23pm UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862/4 "2021-09-29T22:23:01Z")
If you want to limit the number of processes run concurrently to `k`, you can simply use `k` tasks.
```julia
function parallel_run(commands; ntasks = Sys.CPU_THREADS)
request = Channel{Cmd}() do request
for cmd in commands
put!(request, cmd)
end
end
@sync for _ in 1:ntasks
@async try
foreach(run, request)
finally
close(request) # shutdown on error
end
end
end
```
For more techniques like this, see: [Concurrency patterns for controlled parallelisms](https://juliafolds.github.io/data-parallelism/tutorials/concurrency-patterns/)
Also, as baggepinnen mentioned, don’t use `cd` since it mutates the global state. You can use `setdnev(cmd; dir = ...)` to set the directory for each command:
```julia
parallel_run([
`pwd`,
setenv(`pwd`; dir = ".."),
setenv(`pwd`; dir = "/tmp"),
])
```
---
### Author:  [@RCHG](https://discourse.julialang.org/u/RCHG)
#### Post date: [September 30, 2021, 1:37pm UTC](https://discourse.julialang.org/t/how-to-launch-several-run-cmd-in-parallel/68862/5 "2021-09-30T13:37:53Z")
Thank you @tkf@sylvaticus@baggepinnen
All of these comments are solutions! I was not aware about the problem with the global state, this explain why my initial tentative was not working. At this moment @sylvaticus 's solution should work for my setting directly. And, as soon as, I translate some stuff to Julia, then the other solutions are going to be great. Thank you also for the link.