# How to solve error while running parallel sampling in Julia with Turing and Distributed package?

**URL:** <https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551>\
**Category:** Performance\
**Tags:** distributed, optimization, turing\
**Created:** [February 13, 2020, 2:18am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551 "2020-02-13T02:18:38Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manu\_Francis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/manu_francis/32/10010_2.png) [@Manu\_Francis](https://discourse.julialang.org/u/Manu_Francis)\
**Post date:** [February 13, 2020, 2:18am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/1 "2020-02-13T02:18:38Z")

</div>

Hi,  
I am trying to run samplers in parallel by using example code in Turing documentation ([Parallel Sampling](https://turing.ml/dev/docs/using-turing/advanced)). When I run the following example, I am getting an error. The example is as follows:

```julia
using Distributed
addprocs(4)
@everywhere using Turing, Random

# Set the progress to false to avoid too many notifications.
@everywhere turnprogress(false)

# Set a different seed for each process.
for i in procs()
    @fetch @spawnat i Random.seed!(rand(Int64))
end

# Define the model using @everywhere.
@everywhere @model gdemo(x) = begin
    s ~ InverseGamma(2, 3)
    m ~ Normal(0, sqrt(s))
    for i in eachindex(x)
        x[i] ~ Normal(m, sqrt(s))
    end
end

# Sampling setup.
num_chains = 4
sampler = NUTS(0.65)
model = gdemo([1.2, 3.5])

# Run all samples.
chns = reduce(chainscat, pmap(x->sample(model, sampler, 1000), 1:num_chains))

```

and the error as follows:

```julia
On worker 2:
UndefVarError: model not defined
#21 at D:\JULIA\TURING_LSPECT\test_parallel.jl:28
#108 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:294
run_work_thunk at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:79
macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:294 [inlined]
#107 at .\task.jl:333
(::Base.var"#732#734")(::Task) at asyncmap.jl:178
foreach(::Base.var"#732#734", ::Array{Any,1}) at abstractarray.jl:1920
maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at asyncmap.jl:178
wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#208#211"{WorkerPool}, ::Function, ::UnitRange{Int64}) at asyncmap.jl:154
#async_usemap#717(::Function, ::Nothing, ::typeof(Base.async_usemap), ::Distributed.var"#192#194"{Distributed.var"#192#193#195"{WorkerPool,var"#21#22"}}, ::UnitRange{Int64}) at asyncmap.jl:103
(::Base.var"#kw##async_usemap")(::NamedTuple{(:ntasks, :batch_size),Tuple{Distributed.var"#208#211"{WorkerPool},Nothing}}, ::typeof(Base.async_usemap), ::Function, ::UnitRange{Int64}) at none:0
#asyncmap#716 at asyncmap.jl:81 [inlined]
#asyncmap at none:0 [inlined]
#pmap#207(::Bool, ::Int64, ::Nothing, ::Array{Any,1}, ::Nothing, ::typeof(pmap), ::Function, ::WorkerPool, ::UnitRange{Int64}) at pmap.jl:126
pmap(::Function, ::WorkerPool, ::UnitRange{Int64}) at pmap.jl:101
#pmap#217(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(pmap), ::Function, ::UnitRange{Int64}) at pmap.jl:156
pmap(::Function, ::UnitRange{Int64}) at pmap.jl:156
top-level scope at test_parallel.jl:28

```

How can I solve this error? My julia version is _1.3_ and turing is _Turing v0.8.3_

Thanks in Advance !  
Manu

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [February 13, 2020, 3:13am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/2 "2020-02-13T03:13:04Z")

</div>

Hi!

This works:

```julia
using Distributed
addprocs(4)
@everywhere using Turing, Random

# Set a different seed for each process.
for i in procs()
    fetch(@spawnat i Random.seed!(rand(Int64)))
end

@everywhere begin
    # Set the progress to false to avoid too many notifications.
    turnprogress(false)

    # Define the model.
    @model gdemo(x) = begin
        s ~ InverseGamma(2, 3)
	m ~ Normal(0, sqrt(s))
	for i in eachindex(x)
	    x[i] ~ Normal(m, sqrt(s))
	end
    end

    # Sampling setup.
    alg = NUTS(0.65)
    model = gdemo([1.2, 3.5])
    chn = sample(model, alg, 100, save_state = false)
end

chns = mapreduce(chainscat, procs()) do i
    fetch(@spawnat i chn)
end

```

Note that with `addprocs(4)` you actually have 5 parallel processes running.

@cpfiffer it seems that `save_state = true` leads to a similar serialization error as in [Loading saved Chain object fails · Issue #1091 · TuringLang/Turing.jl · GitHub](https://github.com/TuringLang/Turing.jl/issues/1091). `pmap` with closures and even `@spawnat` with local variables which should get sent over seem to be causing issues the first time I run them, but they work the second time. This seems like a Julia issue.

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [February 13, 2020, 4:10am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/3 "2020-02-13T04:10:14Z")

</div>

I’ll just open a PR to disable the default state saving. It’s causing an awful lot of trouble.

---

<div class="post-metadata">

**Author:** ![Manu\_Francis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/manu_francis/32/10010_2.png) [@Manu\_Francis](https://discourse.julialang.org/u/Manu_Francis)\
**Post date:** [February 13, 2020, 4:27am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/4 "2020-02-13T04:27:32Z")

</div>

@mohamed82008 : Thanks a lot for your reply. This solution works most of the times. But sometime, when I run the following section in above code, I am getting error.

```julia
for i in procs()
    fetch(@spawnat i Random.seed!(rand(Int64)))
end

```

The error is as follows.

```julia
On worker 2:
DomainError with -4990743955205970348:
`n` must be non-negative.
make_seed at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Random\src\RNGs.jl:266
seed! at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Random\src\RNGs.jl:289 [inlined]
#9 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\macros.jl:87
#105 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:290
run_work_thunk at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:79
run_work_thunk at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\process_messages.jl:88
#98 at .\task.jl:333
#remotecall_fetch#145 at remotecall.jl:390 [inlined]
remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at remotecall.jl:382
#remotecall_fetch#148 at remotecall.jl:417 [inlined]
remotecall_fetch at remotecall.jl:417 [inlined]
call_on_owner at remotecall.jl:490 [inlined]
fetch(::Future) at remotecall.jl:529
top-level scope at test_parallel.jl:7

```

Thanks again for spending your valuable time to solve my issue.

Thanking you,  
Manu

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [February 13, 2020, 4:34am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/5 "2020-02-13T04:34:08Z")

</div>

Yes that’s a bug in the docs, use `abs(rand(Int))`. And while you are at it, try making a PR to fix the docs.

---

<div class="post-metadata">

**Author:** ![Manu\_Francis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/manu_francis/32/10010_2.png) [@Manu\_Francis](https://discourse.julialang.org/u/Manu_Francis)\
**Post date:** [February 13, 2020, 4:42am UTC](https://discourse.julialang.org/t/how-to-solve-error-while-running-parallel-sampling-in-julia-with-turing-and-distributed-package/34551/6 "2020-02-13T04:42:08Z")

</div>

@mohamed82008: Now everything works fine… thank you 🙂