# Distributed Computing without passing large Data Structure

**URL:** <https://discourse.julialang.org/t/distributed-computing-without-passing-large-data-structure/90469>\
**Category:** General Usage\
**Tags:** parallel, distributed\
**Created:** [November 18, 2022, 5:35pm UTC](https://discourse.julialang.org/t/distributed-computing-without-passing-large-data-structure/90469 "2022-11-18T17:35:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![astrobc1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/astrobc1/32/45632_2.png) [@astrobc1](https://discourse.julialang.org/u/astrobc1)\
**Post date:** [November 18, 2022, 5:35pm UTC](https://discourse.julialang.org/t/distributed-computing-without-passing-large-data-structure/90469/1 "2022-11-18T17:35:09Z")

</div>

I’m trying to use distributed computing without having to pass a large Vector to each worker. Each worker only needs a single item from the Vector. I should also clarify that I cannot use SharedArrays (at least to my knowledge) because for my actual code, the Vector is filled with complex objects (not bits as done here).

Example below:

```julia
using Distributed

addprocs(4)

@everywhere begin
    function expensive_function(y)
        return y * 2 + 1.0
    end
    struct Container
        x::Vector{Float64}
    end
end

function run_parallel(container)
    results = pmap(1:length(container.x)) do i
        expensive_function(container.x[i])
    end
    return results
end

n = 500
x = ones(n^3)
container = Container(x)
run_parallel(container)

```

The container is unnecessary but is closer to how my actual code is setup. When running this code it is clear that a copy of container is passed to each worker, when in reality `expensive_function` only needs a single entry at a time.

I know I can use `pmap(iterator, distributed=false)` but with that option, my code starts crashing within a try/catch block. So first I’d like to exhaust any options with distributed=true.

Thanks!

---

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [November 18, 2022, 6:42pm UTC](https://discourse.julialang.org/t/distributed-computing-without-passing-large-data-structure/90469/2 "2022-11-18T18:42:49Z")

</div>

Have you tried (p)mapping over the elements of the container array? I.e., replace

```julia
pmap(1:length(container.x)) do i

```

with

```julia
pmap(container.x) do xi

```

(I don’t know if this will actually avoid copying the container to all workers, but it might be worth a try.)

---

<div class="post-metadata">

**Author:** ![astrobc1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/astrobc1/32/45632_2.png) [@astrobc1](https://discourse.julialang.org/u/astrobc1)\
**Post date:** [November 18, 2022, 9:03pm UTC](https://discourse.julialang.org/t/distributed-computing-without-passing-large-data-structure/90469/3 "2022-11-18T21:03:54Z")

</div>

That appears to work thank you!

And as a side note - this behavior seems different from [`@distributed`](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.@distributed) where the iterator is copied to each worker. Interesting, but glad it works.

More efficient code for completeness:

```julia
using Distributed

addprocs(4)

@everywhere begin
    function expensive_function(y)
        return y * 2 + 1.0
    end
    struct A
        x
    end
end

function run_parallel(avec)
    results = pmap(avec) do ai
        expensive_function(ai.x)
    end
    return results
end

function make_avec(n)
    avec = Vector{A}(undef, n)
    for i=1:length(avec)
        avec[i] = A(i)
    end
    return avec
end

n = 500^3
avec = make_avec(n)
run_parallel(avec)

```