`pmap` with shared arrays in side a custom type?

General Usage
1

Hi all,
I have got the following example problem:

  • I have a custom type with a shared array
  • I want to distribute computation among n workers
  • My computation requires me to know which job I am running (i.e. the first job is different from the second etc)
module bk
type Foo
    arr::SharedArray
    function Foo(n)
	    this=new()
	    this.arr = SharedArray(Float64,(n,1),pids=workers(),init=S->S[Base.localindexes(S)] = rand())
	    return this
    end
end

function sum(f::Foo,idx::UnitRange)
	# idx is the id of jobs that this worker has to do
	sum(f.arr[idx])		# for example
end
end

then I include my code on all workers

julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> @everywhere include("src/bk.jl")

julia> f = bk.Foo(1000)
bk.Foo([0.465832; 0.465832; … ; 0.501914; 0.501914])

julia> js = bk.jobdist(length(workers()),1000)
2-element Array{UnitRange{Int64},1}:
 1:500   
 501:1000

julia> pmap(x->bk.sum(f,x),js)
ERROR: On worker 2:
UndefVarError: f not defined

I think I understand why this is not working - f was never defined on the workers. But how could I achieve this? i want to create a Foo only in one place (on the master) - Foo should contain the shared arrays and I want to pass the Foo around. any ideas?

2

Perhaps the discussion here is helpful?

https://groups.google.com/forum/#!topic/julia-users/9Wml9MfrZQg

3

yes - very helpful. in fact, that solved this particular problem for me. I will post another question now about memory allocation.