# Using pmap within a package

**URL:** <https://discourse.julialang.org/t/using-pmap-within-a-package/131627>\
**Category:** General Usage\
**Tags:** distributed, pmap\
**Created:** [August 15, 2025, 1:25pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627 "2025-08-15T13:25:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![veddox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/veddox/32/45227_2.png) [@veddox](https://discourse.julialang.org/u/veddox)\
**Post date:** [August 15, 2025, 1:25pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/1 "2025-08-15T13:25:34Z")

</div>

I’ve been trying to wrap my head around how to use `pmap()` properly within a package, but I’m still a bit confused and am starting to suspect my original plan may not work.

For context: I’m working on a process-based ecological model that is computationally quite heavy (multiple hours per simulation run). A typical run follows this workflow:

1. read in configuration file in `getsettings()`
2. based on this, `initialise()` the model object containing all state
3. carry out the simulation using the model object in `simulate()`

However, we frequently have to test different parameter combinations. So that we don’t have to create a separate configuration file for every single run, I have implemented a feature that allows us to define multiple values for a parameter in the configuration file. The `getsettings()` function recognises this, and signals to the `initialise()` and `simulate()` functions that they need to create and run multiple model objects all in one go (one for each combination of parameter values).

As long as I am doing this simply with `map()` it works just fine, but of course it would be nice to parallelise this to speed things up. Since there is no global state, this should be doable. So I thought I would just have to change the `map()` to `pmap()` and add some additional processes based on a configuration parameter:

```julia
function initialise(configfile::String=PARAMFILE)
    settings = getsettings(configfile)
    scanparams = settings["internal.scanparams"]
    if isempty(scanparams)
        initmodel(settings)
    else
        addprocs(settings["core.processors"]-1)
        pmap(initmodel, paramscan(settings, scanparams))
    end
end

function simulate(configfile::String=PARAMFILE)
    models = initialise(configfile)
    isa(models, Vector) ? 
        pmap(simulate!, models) :
        simulate!(models)
end

```

However, this gives me an error: `ERROR: LoadError: On worker 2: KeyError: key Base.PkgId(Base.UUID("039acd1d-2a07-4b33-b082-83a1ff0fd136"), "Persefone") not found`, which I assume means that the worker processes don’t have access to the package I’m working in. I’ve tried fixing this with `@everywhere include("src/Persefone.jl")` before the `addprocs()` call above, but this just generates other errors. So I’m not sure how this could be made to work.

I am also starting to wonder whether `pmap()` is actually intended to be used within a package like this? (For instance, would this require me to prefix every one of our dozens of functions with `@everywhere`?) Or is `pmap()` rather intended to be used in an external script that loads a package and then does something with it?

---

<div class="post-metadata">

**Author:** ![bgroenks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgroenks/32/21784_2.png) [@bgroenks](https://discourse.julialang.org/u/bgroenks)\
**Post date:** [August 15, 2025, 2:43pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/2 "2025-08-15T14:43:00Z")

</div>

I think an MWE would go a long way here.

In general, yes, it is tricky using `pmap` inside of functions because you have to make sure that all of the necessary context is serialized to the remote worker. But it is possible, see for example the implementation of `solve_batch` for `EnsembleProblem` in [SciMLBase](https://github.com/SciML/SciMLBase.jl/blob/d401c2802ac63dea3b0edea6eeefdf38c0cf8f91/src/ensemble/basic_ensemble_solve.jl#L259).

> [@veddox](#):
>
> I’ve tried fixing this with `@everywhere include("src/Persefone.jl")` before the `addprocs()` call above, but this just generates other errors.

Could you be more specific here? What other errors does it generate?

---

<div class="post-metadata">

**Author:** ![veddox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/veddox/32/45227_2.png) [@veddox](https://discourse.julialang.org/u/veddox)\
**Post date:** [August 15, 2025, 4:05pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/3 "2025-08-15T16:05:01Z")

</div>

Hi, thanks for the quick answer! I’m afraid I was struggling to come up with a MWE, because of all the different things going on in the code (and me not being sure which of them are important or not).

However, I realised that putting the call to `pmap()` in a wrapper script would not actually be that difficult, even if it is not quite as nice-to-use as I would have wished. So I wrote the following script, which works fine:

```julia
using Distributed

@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere using Persefone

function parallel_simulate(configfile::String=Persefone.PARAMFILE)
    settings = Persefone.getsettings(configfile)
    scanparams = settings["internal.scanparams"]
    !isempty(scanparams) && 
    delete!(settings, "internal.scanparams")
    if isempty(scanparams)
        Persefone.initmodel(settings) |> simulate!
    else
        combinations = Persefone.paramscan(settings, scanparams)
        pmap(combinations) do c
            Persefone.initmodel(c) |> simulate!
        end
    end
end

@time parallel_simulate()

```

If anybody still has any tips about using `pmap()` in the context of a package, I’d still be interested to hear them. But otherwise this has solved the pressing issue for now.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 15, 2025, 4:49pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/4 "2025-08-15T16:49:57Z")

</div>

It is quite tricky and annoying to use Distributed.jl in packages. This post could help: [Distributed computing for functions in scripts inside a local module? - #5 by dave.f.kleinschmidt](https://discourse.julialang.org/t/distributed-computing-for-functions-in-scripts-inside-a-local-module/96729/5) and here is a package that does use Distributed: [GitHub - beacon-biosignals/OndaBatches.jl: Local and distributed batch loading for Onda datasets](https://github.com/beacon-biosignals/OndaBatches.jl). When Distributed creates workers it does not activate the same Julia environment so they don’t have access to the source code; `remotecall_eval` is one way to activate the environment and load the right code. Instead of `include`ing code on the workers you should activate the package and load it with `using`.

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [August 15, 2025, 6:36pm UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/5 "2025-08-15T18:36:56Z")

</div>

I often use [GitHub - tkf/ThreadsX.jl: Parallelized Base functions](https://github.com/tkf/ThreadsX.jl) because it is easier. Maybe that would work for your problem.

---

<div class="post-metadata">

**Author:** ![veddox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/veddox/32/45227_2.png) [@veddox](https://discourse.julialang.org/u/veddox)\
**Post date:** [August 16, 2025, 8:34am UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/6 "2025-08-16T08:34:14Z")

</div>

Ah, the link to that answer looks quite useful! And good to know about `remotecall_eval`, that hadn’t popped up in anything I’d read so far. I’ll take another look at this, maybe I can do something similar to my original plan after all.

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [August 16, 2025, 9:26am UTC](https://discourse.julialang.org/t/using-pmap-within-a-package/131627/7 "2025-08-16T09:26:40Z")

</div>

Just want to point out that DrWatson.jl has some great features that may be helpful for your situation. I’ve found it extremely helpful for establishing sets of parameters, running simulations, saving results, organizing simulations, etc.