# @Distributed: On worker 2 UndefVarError {{Module}}

**URL:** <https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773>\
**Category:** General Usage\
**Tags:** question\
**Created:** [July 9, 2020, 6:32am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773 "2020-07-09T06:32:57Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ok\_ok](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ok_ok/32/15450_2.png) [@ok\_ok](https://discourse.julialang.org/u/ok_ok)\
**Post date:** [July 9, 2020, 6:32am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/1 "2020-07-09T06:32:58Z")

</div>

Hello,  
i want to run a `for loop` in parallel but using `Threads.threads` just made the Programm slower so i wanted to try `@distributed`

```julia
module MyModule
export start, loop_inner, rec
function start()
    return rec(...)
end

@inline function loop_inner(...)
    ...
end
function rec(...)
    ...
    results = zeros(N)
    @sync @distributed for idx in 1::N
        results[idx] = MyModule.loop_inner(...)
    end
    return sum(results)
    ...
end

```

When i just use`@distributed` without `@sync` in front of it - im getting no `UndefVarError's` but `sum(results)` will always be `0`. When i try to use `@sync @distributed` im getting:

```julia
ERROR: LoadError: TaskFailedException:
On worker 2:
UndefVarError: MyModule not defined

```

My Main looks something like this:

```julia
using Distributed
cwd = pwd()
@Distributed.everywhere push!(LOAD_PATH,"pwd")
@Distributed.everywhere include("MyModule.jl")
@Distributed.everywhere using .MyModule

workers = Threads.nthreads() # = 8
addprocs(workers, 
            #max_parallel = 8,
            restrict=true, 
            enable_threaded_blas=true,
            exeflags=`--optimize=3 --inline=yes --check-bounds=no --math-mode=fast`, 
            topology=:master_worker)

 LangfordParallel.start()

```

`$JULIA_NUM_THREADS=8 julia -L MyModule.jl main.jl -p=8`  
Code: [https://github.com/masterholdy/julia-threading-question](https://github.com/masterholdy/julia-threading-question)

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 9, 2020, 9:31am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/2 "2020-07-09T09:31:57Z")

</div>

I have downloaded your code into `c:\temp` and made it run by changing `main_langford.jl`:

This

```julia
cwd = pwd()
@Distributed.everywhere push!(LOAD_PATH,"pwd")

```

doesn’t make much sense. You don’t want to have the string “pwd” in the LOAD\_PATH. I failed for now to do it right, but the following line helped as a workaround for now:

```julia
@everywhere cd("c:\\temp")

```

Which puts every worker into the desired working directory.

I commentet out the topology as it doesn’t work for me:

```julia
addprocs(workers, 
            #max_parallel = 8,
            restrict=true, 
            enable_threaded_blas=true,
            exeflags=`--optimize=3 --inline=yes --check-bounds=no --math-mode=fast`, 
            #topology=:master_worker
)

```

and put it directly after

```julia
using Distributed

```

So the complete changed part of your `main_langford.jl` is now:

```julia
#include("LangfordParallel.jl")
using Distributed
workers = Threads.nthreads()
addprocs(workers, 
            #max_parallel = 8,
            restrict=true, 
            enable_threaded_blas=true,
            exeflags=`--optimize=3 --inline=yes --check-bounds=no --math-mode=fast`, 
            #topology=:master_worker
)

#cwd = pwd()
@everywhere cd("c:\\temp")
#@Distributed.everywhere include("langford.jl")
@Distributed.everywhere include("LangfordParallel.jl")
@Distributed.everywhere using .LangfordParallel
#@Distributed.everywhere include("LangfordBinary.jl")
#@Distributed.everywhere using Distributed
#using Revise, Profile, PProf
#addprocs(8)

```

Now starting the REPL (1.4.2, Windows 10 64bit):

```julia
julia> cd("c:\\temp")

julia> include("main_langford.jl")
L(s, n) = L(2, 12):
-------------------------------
Start w/: 1 Thread(s)
ERGEBNIS-Sequential (Julia) :0.0
Elapsed time in microseconds : 15311.017199 ms
-------------------------------

```

Running Julia with `JULIA_NUM_THREADS=8` and doing the same brings:

```julia
julia> cd("c:\\temp")

julia> include("main_langford.jl")
L(s, n) = L(2, 12):
-------------------------------
Start w/: 8 Thread(s)

```

and still running (takes very long it seems).

So, your code has quite some issues and I don’t have the time now to do it all right. But I hope that the above changes will help you to do the next steps.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 9, 2020, 9:48am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/3 "2020-07-09T09:48:19Z")

</div>

What I forgot to mention:

It seems you are mixing Threads and multipe CPUs in your code. You may read [Parallel Computing · The Julia Language](https://docs.julialang.org/en/v1/manual/parallel-computing/) again. In general: modern CPUs have multiple Cores, you use them with Distributed. A single core does have multiple threads, thats where multi-threading comes into play.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 9, 2020, 11:25am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/5 "2020-07-09T11:25:12Z")

</div>

First I would suggest, decide, which parallel paradigm to use first: threads OR cores. If you want to start with threads remove Distributed, @ everywhere and addprocs and so on. Stick on the Threads documentation [Parallel Computing · The Julia Language](https://docs.julialang.org/en/v1/manual/parallel-computing/#man-multithreading-1)

(Of course your code should produce the desired result without any paralell execution at the very first!)

> [@ok\_ok](#):
>
> i thought a single core has 2 Threads if you have SMT/Hyper-Threading CPU?.  
> (8 Cores → 16 Threads)

Yes for the typicall CPU, there may be CPUs with more than 2 threads per core, don’t know, not important for us now.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 9, 2020, 11:45am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/7 "2020-07-09T11:45:24Z")

</div>

Yes, thats what I think too.

What I try to say is: Using threads doesn’t need Distributed, everywhere macro, addproc and all those things which belong to multi processes. You want (I guess) multiple threads in a single process (for the start). This is described in  
[https://docs.julialang.org/en/v1/manual/parallel-computing/#man-multithreading-1](https://docs.julialang.org/en/v1/manual/parallel-computing/#man-multithreading-1)  
and when you scroll down it stops with the headline

# [Multi-Core or Distributed Processing](https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Core-or-Distributed-Processing-1)

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 9, 2020, 1:12pm UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/9 "2020-07-09T13:12:53Z")

</div>

I can’t see a question anymore, but still it seems you are just trying this and that quite randomly.  
(e.g. @inbounds should not pop up here).

The docs are quite good to get the basics about parallel execution.

As we were talking about threads and are heading for a 2CPU/72Thread machine I have put up an example using threads. And because I wasn’t sure if threads are meant to be created on every existing core or just on the cores set available (with the -p parameter when calling Julia). I didn’t found a fast answer in the docs so I just put some code together:

```julia
using Random, BenchmarkTools

N=10_000_000
NThreads=8
MersenneTwister(0); #seeding the RNG
a=[rand(Int,N) for x = 1:NThreads]

function f_threaded(a,N,NThreads)
	s=zeros(NThreads)
	indices=collect(1:(N-1))
	randperm!(indices)
	for k in 1:10
		Threads.@threads for i in 1:NThreads
			for j in indices
				s[i]+=a[i][j]*a[i][j+1]
			end
		end
	end
	s
end
function f(a,N,NThreads)
	s=zeros(NThreads)
	indices=collect(1:(N-1))
	randperm!(indices)
	for k in 1:10
		for i in 1:NThreads
			for j in indices
				s[i]+=a[i][j]*a[i][j+1]
			end
		end
	end
	s
end
@btime f(a,N,NThreads)
@btime f_threaded(a,N,NThreads)

Threads.nthreads()

```

On a windows CMD:

```julia
C:\temp>set JULIA_NUM_THREADS=16

C:\temp>julia.exe

```

The outcome is:

```julia
julia> @btime f(a,N,NThreads)
  11.607 s (3 allocations: 76.29 MiB)
8-element Array{Float64,1}:
 -3.88638726543992e23
 -4.138778296184118e23
 -2.967358655409068e23
  1.5760206471213304e23
 -1.7625091619973682e23
  2.07202393944643e23
  2.6818004920701988e23
 -9.87477346267389e22

julia> @btime f_threaded(a,N,NThreads)
  5.934 s (466 allocations: 76.36 MiB)
8-element Array{Float64,1}:
 -3.8863872654405564e23
 -4.138778296183166e23
 -2.9673586554094026e23
  1.5760206471210798e23
 -1.762509161997028e23
  2.0720239394459922e23
  2.681800492070774e23
 -9.874773462673677e22

julia> Threads.nthreads()
8

```

Twice as fast using threads. And:

 ![cpus](https://global.discourse-cdn.com/julialang/original/3X/a/f/afaccd005a5a3f0c64b3fe0d204e059b7be5b0d7.jpeg)

Using `set JULIA_NUM_THREADS=16` results in

```julia
julia> Threads.nthreads()
8

```

and all 8 cores are 100% in use. By the way, the little nodge during the 100% usage was taking the screenshot of the window 🙂

So, all you need is Threads.@threads and you need to check for side effects and race conditions, everything very good covered in the docs. Of course your algorithm should be put into good shape for parallelism. I haven’t viewed what you are doing in your code, because you are still at the very beginning to make your code run and produce something and not only errors.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [July 13, 2020, 6:43pm UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/11 "2020-07-13T18:43:34Z")

</div>

> [@ok\_ok](#):
>
> ```julia
> function sum_of_sum_parallel_1(a, b)
> taskL = Threads.@spawn sum(a)
> l = fetch(taskL)
> r = sum(b)
> return l + r
> end
> 
> ```

You want to `fetch` _after_ calling `sum(b)`. That is,

```julia
function sum_of_sum_parallel_1(a, b)
    taskL = Threads.@spawn sum(a)
    r = sum(b)
    l = fetch(taskL)
    return l + r
end

```

In `parallelSum`, you want to put the `@threads` on the `for thread_idx in 1:numberThreads`, not the inner loop:

```julia
    Threads.@threads for thread_idx in 1:numberThreads
       for idx in thread_idx:numberThreads:problemSize
            results[thread_idx] += a[idx]
        end
    end

```

Alternatively, you could use:

```julia
    @sync for thread_idx in 1:numberThreads
       Threads.@spawn for idx in thread_idx:numberThreads:problemSize
            results[thread_idx] += a[idx]
        end
    end

```

The `@threads` macro _splits up_ a `for` loop to run on multiple threads; `@spawn` creates an _individual task_ that can run on any thread.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 14, 2020, 7:34am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/14 "2020-07-14T07:34:47Z")

</div>

Strange, I can’t confirm:

```julia
C:\Users>set JULIA_NUM_THREADS=12

julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 5 3600 6-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, znver1)
Environment:
  JULIA_NUM_THREADS = 12

using Random, BenchmarkTools

N=100_000_000
NThreads=12
MersenneTwister(0); #seeding the RNG
a=rand(Float64,N)

function f_threaded(a, problemSize, numberThreads = Threads.nthreads())
    results = zeros(numberThreads)
    Threads.@threads for thread_idx in 1:numberThreads
        @simd for idx in thread_idx:numberThreads:problemSize
            @inbounds results[thread_idx] += a[idx]
        end
    end
    return sum(results)
end
function f(a, problemSize, numberThreads = Threads.nthreads())
    results = zeros(numberThreads)
    for thread_idx in 1:numberThreads
        for idx in thread_idx:numberThreads:problemSize
            results[thread_idx] += a[idx]
        end
    end
    return sum(results)
end

julia> @btime f(a,N,NThreads)
  434.296 ms (2 allocations: 192 bytes)
4.99935545816905e7

julia> @btime f_threaded(a,N,NThreads)
  113.651 ms (65 allocations: 10.13 KiB)
4.99935545816905e7

```

How do you benchmark? Can you provide complete MWE(=minimal working example) code?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 14, 2020, 10:07am UTC](https://discourse.julialang.org/t/distributed-on-worker-2-undefvarerror-module/42773/16 "2020-07-14T10:07:29Z")

</div>

If you want to beat `sum` you will have a hard time.  
Have a look at

```julia
@edit sum(a)

```

And you may want to read this discussion: [Implementing parallel sum](https://discourse.julialang.org/t/implementing-parallel-sum/31097)  
There are others also about `sum` and performance, just search for sum.