Distributed Computing without passing large Data Structure

General Usage
,
1

I’m trying to use distributed computing without having to pass a large Vector to each worker. Each worker only needs a single item from the Vector. I should also clarify that I cannot use SharedArrays (at least to my knowledge) because for my actual code, the Vector is filled with complex objects (not bits as done here).

Example below:

using Distributed

addprocs(4)

@everywhere begin
    function expensive_function(y)
        return y * 2 + 1.0
    end
    struct Container
        x::Vector{Float64}
    end
end

function run_parallel(container)
    results = pmap(1:length(container.x)) do i
        expensive_function(container.x[i])
    end
    return results
end

n = 500
x = ones(n^3)
container = Container(x)
run_parallel(container)

The container is unnecessary but is closer to how my actual code is setup. When running this code it is clear that a copy of container is passed to each worker, when in reality expensive_function only needs a single entry at a time.

I know I can use pmap(iterator, distributed=false) but with that option, my code starts crashing within a try/catch block. So first I’d like to exhaust any options with distributed=true.

Thanks!

2

Have you tried (p)mapping over the elements of the container array? I.e., replace

pmap(1:length(container.x)) do i

with

pmap(container.x) do xi

(I don’t know if this will actually avoid copying the container to all workers, but it might be worth a try.)

3

That appears to work thank you!

And as a side note - this behavior seems different from @distributed where the iterator is copied to each worker. Interesting, but glad it works.

More efficient code for completeness:

using Distributed

addprocs(4)

@everywhere begin
    function expensive_function(y)
        return y * 2 + 1.0
    end
    struct A
        x
    end
end

function run_parallel(avec)
    results = pmap(avec) do ai
        expensive_function(ai.x)
    end
    return results
end

function make_avec(n)
    avec = Vector{A}(undef, n)
    for i=1:length(avec)
        avec[i] = A(i)
    end
    return avec
end

n = 500^3
avec = make_avec(n)
run_parallel(avec)