# Error running distributed code inside of a module

**URL:** <https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283>\
**Category:** General Usage\
**Created:** [January 30, 2021, 2:55pm UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283 "2021-01-30T14:55:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [January 30, 2021, 2:55pm UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/1 "2021-01-30T14:55:26Z")

</div>

The following works just fine:

```julia
using Distributed

procs = addprocs(10)

futures = []
for proc in procs
	push!(futures, @spawnat proc sleep(2))
end
for fut in futures
	fetch(fut)
end

```

The following:

```julia
module A
    using Distributed

    procs = addprocs(10)

    futures = []
    for proc in procs
        push!(futures, @spawnat proc sleep(2))
    end
    for fut in futures
        fetch(fut)
    end
end

```

throws

```julia
ERROR: On worker 2:
UndefVarError: A not defined

```

on the line calling `fetch`.

I’m familiar with the need to make sure the right stuff ends up on the worker processes, e.g. making sure to add lines like

```julia
for proc in procs
    @spawnat proc begin
        import Pkg
        Pkg.activate(...)
        import Library1, Library2
    end
end

```

prior to using whatever functionality I need.

But in the problem case above, the worker processes don’t need to know anything at all about the module `A`. My belief was that each process was a fresh copy of Julia, that runs whatever code we send to it – in this case just a sleep – but clearly that mental model isn’t quite accurate.

For context the desire is to create a library function that uses parallelism:

```julia
module MyLibrary
    using Distributed
    function my_parallel_function(num_procs)
        procs = addprocs(num_procs)
        ...
    end
end

```

What’s the appropriate way to do this? Combing through the documentation/stackoverflow/discourse hasn’t turned up much.

CC @MilesCranmer

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [January 30, 2021, 4:05pm UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/2 "2021-01-30T16:05:17Z")

</div>

> [@patrick-kidger](#):
>
> ```julia
> ERROR: On worker 2:
> UndefVarError: A not defined
> 
> ```

in the 2nd case `procs` is a global variable in `Module A`:

```julia
julia> procs
ERROR: UndefVarError: procs not defined

julia> A.procs
10-element Array{Int64,1}:
 2
 3
....

```

BTW: for doing such initialization stuff for a module in distributed code, you should look at ` __init__ `:

```julia
help?> __init__
search: __init__

  __init__

  __init__ () function in your module would executes immediately after the module is
  loaded at runtime for the first time (i.e., it is only called once and only after
....

```

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [January 30, 2021, 4:43pm UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/3 "2021-01-30T16:43:32Z")

</div>

Thanks for your response. The worker processes don’t need access to `procs`, however. And indeed if I make everything local to a function then the same error occurs:

```julia
module A
    using Distributed

    function foo()
        procs = addprocs(10)

        futures = []
        for proc in procs
            push!(futures, @spawnat proc sleep(2))
        end
        for fut in futures
            fetch(fut)
        end
    end
end

A.foo()

```

This intended to be done at runtime, not initialization.

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [January 30, 2021, 7:34pm UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/4 "2021-01-30T19:34:36Z")

</div>

ok, I can only guess that `@spawnat` needs `A` as context for creating a closure to run on asynchronous workers. `Distributed.pmap` does not have that limitation. With …

```julia
module A
    using Distributed

    function foo()
        futures = []
        for proc in procs()
            push!(futures, @spawnat proc myid())
        end
        fetch.(futures)
    end

    function __init__ ()
        myid() == 1 && nprocs() < 11 && addprocs(11-nprocs())
    end 
end

```

the following works:

```julia
julia> include("A.jl")
Main.A

julia> using Distributed

julia> pmap(_->myid(), 1:nprocs())
11-element Array{Int64,1}:
  2
  3
  4
  6
  5
  9
  7
  8
 10
 11
  4

julia> @everywhere include("A.jl")
WARNING: replacing module A.

julia> A.foo()
11-element Array{Int64,1}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11

```

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [January 31, 2021, 12:25am UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/5 "2021-01-31T00:25:02Z")

</div>

Unfortunately that solution requires the line `@everywhere include("A.jl")` called from the REPL/`Main`. The desire is to be able to wrap up these details so that the user need specify no more than something like `MyModule.my_parallel_function(..., number_of_processes=5)`.

As far as I can tell this is impossible. I’ve not been able to use `@spawnat` or `pmap` for this purpose anywhere except from the REPL/`Main`. Once they’re put inside a module, it appears one runs into a catch-22. Calling remote functions requires the containing module in order to run; but making the containing module available requires calling a remote function.

_(To be clear: I’m aware that you can use `@spawnat` or `pmap` inside a module – but only by requiring some outside help, initialising the processes appropriately first. So it’s essentially impossible to use Distributed as an implementation detail inside a library; it always leaks into user-land; becomes tricky to use libaries inside libraries etc.)_

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [January 31, 2021, 3:00am UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/6 "2021-01-31T03:00:03Z")

</div>

The following code seems to work, but I’m not sure how robust this is?

File `mwe.jl`

```julia
module A
	export test
	using Distributed

	function test()
		procs = addprocs(10)
		@everywhere procs begin
			Base.MainInclude.eval(include(@ __FILE__ ))
			Base.MainInclude.eval(using .A)
		end
		futures = []
		for proc in procs
			push!(futures, @spawnat proc println("Hello from $proc"))
		end
		for fut in futures
			fetch(fut)
		end
	end
end

```

File `run.jl`:

```julia
include("mwe.jl")
using .A
test()

```

Output:

```julia
      From worker 9: Hello from 9
      From worker 6: Hello from 6
      From worker 10: Hello from 10
      From worker 7: Hello from 7
      From worker 3: Hello from 3
      From worker 2: Hello from 2
      From worker 4: Hello from 4
      From worker 5: Hello from 5
      From worker 11: Hello from 11
      From worker 8: Hello from 8

```

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [January 31, 2021, 4:20am UTC](https://discourse.julialang.org/t/error-running-distributed-code-inside-of-a-module/54283/7 "2021-01-31T04:20:27Z")

</div>

Amazing, thank you!

For posterity (and those Googling who end up landing here in the future), here’s a short write-up.

After a bit of experimentation, it seems like `@everywhere procs ...`, for the initalisation step, is the secret sauce here. It works when `for proc in procs; @spawnat proc ...; end` does not. I’ve not tried digging into why that is.

By-the-by one needs to include a couple of extra lines to activate the current environment inside the new worker processes. I’ve tested that the following works when installed as a package called `MyPackage`:

```julia
# MyPackage.jl

module MyPackage

import Distributed
import Pkg

function test(procs)
    project_path = splitdir(Pkg.project().path)[1]
    Distributed.@everywhere procs begin
        Main.eval(quote
            import Pkg
            Pkg.activate($$project_path)
			import MyPackage
        end)
    end

    Distributed.pmap(_->println("Hello from " * string(Distributed.myid())), Distributed.WorkerPool(procs), 1:6)
end

function test(num_procs::Integer)
    project_path = splitdir(Pkg.project().path)[1]
    procs = Distributed.addprocs(num_procs)
    try
        test(procs)
    finally
        Distributed.rmprocs(procs)
    end
end

end

```

In each case the complicated bit being the initial set-up on each worker. After that you’re free to use `pmap`, `@spawnat` or `@everywhere` as you prefer.

I’ve not tested every combination of `[my code above, @MilesCranmer's code above] x [environment, no environment] x [package, no package]` and the behaviour in each case. I’m pretty sure sometimes it’s different. YMMV.