# Run external commands in parallel

**URL:** <https://discourse.julialang.org/t/run-external-commands-in-parallel/20268>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [January 30, 2019, 10:49am UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268 "2019-01-30T10:49:53Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 30, 2019, 10:49am UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/1 "2019-01-30T10:49:53Z")

</div>

I have a computation for which the time-consuming part boils down to calling an external program, in particular with `read(cmd, String)` in parallel for 1-5 values of `cmd`.

Should I just construct the commands, and use `pmap(c -> read(c, String), cmds)`?

(Sorry if this is an obvious question, I am not familiar with parallel language mechanisms, and got confused with threads, `asyncmap`, etc).

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [January 30, 2019, 11:51am UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/2 "2019-01-30T11:51:37Z")

</div>

I am not an expert, but I usually use `asyncmap` for this and it works fine.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [January 30, 2019, 12:10pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/3 "2019-01-30T12:10:07Z")

</div>

A simple loop would also work:

```julia
@sync begin
   for i in 1:10
       @async println(read(`echo $i`, String))
   end
end

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 30, 2019, 12:20pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/4 "2019-01-30T12:20:30Z")

</div>

> [@jw3126](#):
>
> I am not an expert, but I usually use `asyncmap` for this and it works fine.

The docs of `?asyncmap` says

> Note
> 
> Currently, all tasks in Julia are executed in a single OS thread co-operatively. Consequently, asyncmap is beneficial only when the mapping function involves any I/O - disk, network, remote worker invocation, etc.

and I am not sure it if applies here as I wait for the output. I will experiment.

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [January 30, 2019, 12:46pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/5 "2019-01-30T12:46:19Z")

</div>

It depends what the task `cmd` does: CPU - intensive → `pmap`, IO intensive → `asyncmap`. If the task uses a resource intensively i.e (disk/network at max throughput) it may be worthwhile running the commands in a serial fashion.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 30, 2019, 12:48pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/6 "2019-01-30T12:48:41Z")

</div>

Thanks, the task is CPU intensive, but running it on multiple cores (from the command line) results in large speed gains.

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [January 30, 2019, 6:03pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/7 "2019-01-30T18:03:13Z")

</div>

AFAICT this doc refers to native julia code only. Each `cmd` will run in an isolated process. So as long as reading the sheer amount of output of each `cmd` is not the bottle neck, you should get good speed.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 30, 2019, 10:32pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/8 "2019-01-30T22:32:25Z")

</div>

The usual picture is that thread is for CPU-bound and async coroutine for I/O-bound. Here, I believe I/O means anything goes through syscall so spawning subprocesses is included, even though the work inside the subprocess is CPU-bound. So, `asyncmap` (or `@async` etc.) seems to be the right choice to me. I expect Julia to handle multiple concurrent `read`s since it uses `libuv` (which communicates with kernel to switch to the I/O that is ready automatically).

As for the `pmap`, I think it uses async coroutines underneath to communicate with Julia sub-/remote processes. So if you spawn a subprocesses inside `pmap`, I think you are adding additional syscalls and it’s likely to slow down your program (though maybe that’s negligible).

Side notes: I think I/O-bound usually means the number of concurrent works is too large compared to the number of threads that the kernel can handle, as in web services. Since a process is more expensive than a thread, if you are launching processes then maybe it’s a bit misleading to call it I/O-bound or intensive. But, as async is more stable API than threads in Julia, why not use async?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [January 31, 2019, 1:30am UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/9 "2019-01-31T01:30:05Z")

</div>

Do you need to exchange data between the running processes? If not, this would work:

```julia
using Distributed

@everywhere job(p) = begin
		run(`echo Process $p started`)
		sleep(4*rand())
		run(`echo Process $p finished`)
		return p
	end

function runem(howmanyprocesses)
	if nworkers() < howmanyprocesses
		addprocs(howmanyprocesses-nworkers())
	end
	results = [] # collect the remote references to be able to extract the results
	for p = workers()
		rref = @spawnat p job(p)
		push!(results, rref)
	end
	for rref = results
		if fetch(rref) <= 0
			@warn "Failure:"
			@show rref
		end
	end
	return true
end

runem(4)

```

---

<div class="post-metadata">

**Author:** ![JanErik](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@JanErik](https://discourse.julialang.org/u/JanErik)\
**Post date:** [February 7, 2019, 3:39pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/10 "2019-02-07T15:39:43Z")

</div>

I am also looking at running external commands in parallel. In my case, there’s no exchange of data between processes. I tried the helpful suggestion provided by PetrKryslUCSD. Running it without specifying the number of processes (without the -p option) I get the following error:  
ERROR: LoadError: On worker 2:  
UndefVarError: #job not defined

If I start julia with -p greater than the target number of processes (argument to runem) it works fine. Trying to understand this, I moved the job function into runem and it works fine irrespective of the -p option:

```julia
using Distributed

function runem(howmanyprocesses)
	if nworkers() < howmanyprocesses
		addprocs(howmanyprocesses-nworkers())
	end
	results = [] # collect the remote references to be able to extract the results
	for p = workers()
		rref = @spawnat p begin
                    run(`echo Process $p started`)
                    my_time = rand()
		    sleep(4*my_time)
		    run(`echo Process $p finished after $my_time`)
		    return p
	        end
		push!(results, rref)
	end
	for rref = results
		if fetch(rref) <= 0
			@warn "Failure:"
			@show rref
		end
	end
	return true
end

```

I’m new to Julia and am trying to get a handle on how scoping works From my naive perspective, it seems that the processes created by addprocs() in the runem function are local to the function.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [February 7, 2019, 6:59pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268/11 "2019-02-07T18:59:22Z")

</div>

Right, I forgot about loading the function `job` on all the workers created with `addprocs()`. Sorry.