# Write a function that stands up and shuts down workers?

**URL:** <https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041>\
**Category:** General Usage\
**Tags:** pmap\
**Created:** [October 21, 2022, 12:23am UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041 "2022-10-21T00:23:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)\
**Post date:** [October 21, 2022, 12:23am UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/1 "2022-10-21T00:23:37Z")

</div>

I have a utility in a package to run a batch of things. I’d like for that batch function to be able to take in an argument like `n_workers`, then stand those workers up, distribute the work, and then remove those new workers. Here’s an example of how things work today, without remote workers:

```julia
module MyUtility # Pretend this is a stand-alone package.

using Random
using Distributed

# The expensive function we want to run many times:
function foo(n)
    return randn(n)
end

# A convenient function to run foo for us a whole bunch of times, reducing the result with
# whatever the user provides.
function batch(g, reduce)
    return pmap(g) do n
        reduce(foo(n))
    end
end

end

# Here's my script to use the above package.

import .MyUtility

# We want to run MyUtility.foo a bunch of times, reducing the results of each run with this:
function bar(draws)
    return sqrt(sum(draws.^2)/length(draws))
end

results = MyUtility.batch(1:100, bar)

@show results

```

This clearly works fine. Further, I’m able to add boilerplate to this to make it work with distributed workers. However, I’d like to reduce the boilerplate that users of the package will need in order for `batch` to complete its job. I would like for something like this to work:

```julia
function batch(g, reduce, n_workers)

    # Get help.
    workers = nothing
    if n_workers > 0
        workers = addprocs(n_workers)
    end

    # Now do the thing.
    results = pmap(g) do n
        reduce(foo(n))
    end

    # Thank our help and release them.
    if n_workers > 0
        rmprocs(workers)
    end

    return results
end

```

However, this clearly won’t work. Those workers don’t know anything about `bar` or anything used by `bar`. Further, if `batch` is to stand up the workers, then we can’t wrap `bar` in `@everywhere` (or rather, we can, but it won’t make it to the right workers). So I’m confused about the right way to do this.

Here’s the “lots of boilerplate example” way to do it, where all of the parallel stuff is pushed on the end user rather than being a convenient part of the utility:

```julia
using Distributed
addprocs(5, exeflags="--project=$(Base.active_project())") # Make sure workers inherit our project!

# Bring in packages.
@everywhere begin
    import .MyUtility
end

# This needs to be a separate @everywhere block for some reason.
@everywhere begin
    function bar(draws)
        return sqrt(sum(draws.^2)/length(draws))
    end
end

results = MyUtility.batch(1:100, bar)

```

Clearly, moving from the “regular” version to a “parallel” version implies a very big reorganization of the user’s code, when all they really intend is to “distribute what I’m doing across more cores”. In fact, the one place where they _intend_ to have a change (“run batch on all my cores”) is the only line that doesn’t change.

Any tips for how to do this? I’ve tried to read all of the related threads but haven’t found a good way.

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [October 21, 2022, 7:55am UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/2 "2022-10-21T07:55:42Z")

</div>

In some of my own code I’ve used `Meta.eval` to call `@everywhere` from inside a function:

```julia
#... Inside a function
Meta.eval("@everywhere using MyUtility")

```

\*\*\*Edit, also make sure that `addprocs` activates the current project so that it has access to `MyUtility`.

p.s. Is there any reason why you can’t use multithreading for your parallelism instead? Something like:

```julia
results = Vector{Any}(undef, length(a))
Threads.@threads for i in 1:length(a)
    results[i] = reduce(foo(a[i]))
end

```

This should have better performance and lower latency and will work as long as `foo` is not mutating any global state, causing a race condition.

---

<div class="post-metadata">

**Author:** ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)\
**Post date:** [October 21, 2022, 6:26pm UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/3 "2022-10-21T18:26:14Z")

</div>

Thanks @jmair. Somehow I was stuck on the idea that I needed `pmap`! **Switching to `Threads.@threads` is exactly what I needed.**

(I wonder why `pmap` doesn’t use threads, or if there’s an equivalent function that does?)

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [October 21, 2022, 6:37pm UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/4 "2022-10-21T18:37:20Z")

</div>

> [@tuckermcclure](#):
>
> (I wonder why `pmap` doesn’t use threads, or if there’s an equivalent function that does?)

Not in `Base` but there are [ThreadsX.map](https://tkf.github.io/ThreadsX.jl/dev/#ThreadsX.map) or [ThreadPools.tmap](https://tro3.github.io/ThreadPools.jl/build/#ThreadPools.tmap-Tuple%7BFunction,Any%7D), for example.

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [October 21, 2022, 9:27pm UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/5 "2022-10-21T21:27:55Z")

</div>

`pmap` is for the `Distributed.jl` library which is for multiprocessing specifically. As @carstenbauer said, there are some packages that give you `pmap` like syntax.

---

<div class="post-metadata">

**Author:** ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)\
**Post date:** [October 22, 2022, 5:25am UTC](https://discourse.julialang.org/t/write-a-function-that-stands-up-and-shuts-down-workers/89041/6 "2022-10-22T05:25:19Z")

</div>

Thanks @carstenbauer . `ThreadPools.tmap` looks like what I was expecting `pmap` to offer, and `tmap` seems like an obvious candidate for `Threads` even. It functions like I expect coming from other languages. Very nice suggestions. Thank you!