How to use CachingPool?

Specific Domains Julia at Scale
1

Do I used the CachingPool incorrectly, or is this a bug?

On Julia 1.0.2:

using Distributed
addprocs(2)
@everywhere f_passall(a,x) = length(x) + a
its = 1:20
bigarr = ones(10^8)
pool = CachingPool(workers())
pmap(pool, x->f_passall(x,bigarr), its)

throws:

MethodError: no method matching iterate(::getfield(Main, Symbol("##9#10")))
Closest candidates are:
  iterate(!Matched::Core.SimpleVector) at essentials.jl:589
  iterate(!Matched::Core.SimpleVector, !Matched::Any) at essentials.jl:589
  iterate(!Matched::ExponentialBackOff) at error.jl:171
  ...
in top-level scope at base\none
in pmap at stdlib\v1.0\Distributed\src\pmap.jl:157
in #pmap#226 at stdlib\v1.0\Distributed\src\pmap.jl:157
in pmap at stdlib\v1.0\Distributed\src\pmap.jl:156
in #pmap#225 at stdlib\v1.0\Distributed\src\pmap.jl:156
in pmap at stdlib\v1.0\Distributed\src\pmap.jl:101
in #pmap#215 at stdlib\v1.0\Distributed\src\pmap.jl:126
in  at base\none
in #asyncmap#667 at base\asyncmap.jl:81
in  at base\none
in #async_usemap#668 at base\asyncmap.jl:103
in wrap_n_exec_twice at base\asyncmap.jl:154
in maptwice at base\asyncmap.jl:162
in map at base\abstractarray.jl:2013
in collect at base\array.jl:619
in iterate at base\generator.jl:44 
in iterate at base\iterators.jl:320 
in zip_iterate at base\iterators.jl:300
2

I think you might have the argument order wrong:

Have you tried pmap(x->f_passall(x,bigarr), pool, its) ?

See Distributed Computing · The Julia Language

Xref: https://github.com/JuliaLang/julia/issues/30332

3

The second argument is a WorkerPool, which defaults to the pool of idle workers.

4

You are right, the CachingPool is a subtype of an AbstractWorkerPool! Thank you!

It might be used incorrectly (compared to Julia 1.0.2) here, here, here, and here.

5

pmap arguments were re-ordered to allow do-block syntax ( reorder pmap arguments to allow do-block syntax by mweastwood · Pull Request #26783 · JuliaLang/julia · GitHub ).

Using v0.7 gives you deprecation message:

Julia-0.7.0> pmap(pool, x->f_passall(x,bigarr), its)
┌ Warning: `pmap(p::AbstractWorkerPool, f, c; kwargs...)` is deprecated, use `pmap(f, p, c; kwargs...)` instead.
│   caller = top-level scope at none:0
└ @ Core none:0
1 Like