How to include a Distributed into a package?

General Usage
1

I am writing a package where at some point a big loop appears.
What is the proper way to allow using the Distributed.jl package to replace map by pmap?

  1. Should we do like that:
module MyPackage
using Distributed

### code
fit_function() = pmap(long function)
### code
export fit_function
end

and then

using MyPackage
fit_function()

when we are using personal laptop with no multiprocessor.
On a cluster

using Distributed
@everywhere using MyPackage
addproc!(10)
fit_function()
  1. is there another preferable architecture like two different packages with/without Distributed pmap/map ?
  2. maybe always do using Distributed but defining two functions fit_function() with pmap and another one pfit_function() with pmap.

What is the best way to go in terms of

  • “code”
  • Julia spirtit,
  • Dependency,
  • Performances (from what I recall pmap was slightly longer than map when used on one proc).
2

I still wonder how one can allow pmap on some operations conditionally to loading Distributed.jl
Would the new package extension be suitable for that?

3

To anyone interested and maybe related to some other questions, what works for me is to add

module MyPkg
using Distributed
f(x) = ...
function big(...)
  ....
  return pmap(f, iterator)
end

export big
end

Then outside the pkg I just need to do

using Distributed
addprocs(10)
using MyPkg
@everywhere using MyPkg
big(...) # works and is faster!

It is very simple to modify existing code to make it distributed that way.
If I don’t want distributed, I just do not load using Distributed and pmap performs as a regular map.