# Parallel fetch into pre-allocated memory?

**URL:** <https://discourse.julialang.org/t/parallel-fetch-into-pre-allocated-memory/3238>\
**Category:** General Usage\
**Created:** [April 16, 2017, 3:30am UTC](https://discourse.julialang.org/t/parallel-fetch-into-pre-allocated-memory/3238 "2017-04-16T03:30:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![samtkaplan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samtkaplan/32/2000_2.png) [@samtkaplan](https://discourse.julialang.org/u/samtkaplan)\
**Post date:** [April 16, 2017, 3:30am UTC](https://discourse.julialang.org/t/parallel-fetch-into-pre-allocated-memory/3238/1 "2017-04-16T03:30:27Z")

</div>

Hello,

I’m using `fetch` to communicate large arrays between nodes on a compute cluster. This results in more memory allocation than I would like. Is there an in-place version of `fetch` that uses pre-allocated memory? Here is an example to illustrate:

```julia
addprocs(2)

function main()
    f1 = @spawnat workers()[1] rand(1000)
    f2 = @spawnat workers()[2] rand(1000)
    x = zeros(1000)

    x[:] = fetch(f1) # I would like to do something like fetch!(f1,x)
    y = sum(x)
    x[:] = fetch(f2)
    z = sum(x)

    @show y,z
end

main()

```

If I understand correctly, both calls to `fetch` will allocate memory for an array that is the same size as `x`. Subsequently this memory is copied from the `Future` to `x`. This extra allocation and copy is what I want to try and avoid.

Thanks for taking time to read this, and for any ideas towards a solution?

Sam

---

<div class="post-metadata">

**Author:** ![amit.murthy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amit.murthy/32/47_2.png) [@amit.murthy](https://discourse.julialang.org/u/amit.murthy)\
**Post date:** [April 17, 2017, 11:39am UTC](https://discourse.julialang.org/t/parallel-fetch-into-pre-allocated-memory/3238/2 "2017-04-17T11:39:43Z")

</div>

This would be a good feature to have.

Currently I think you will need to manage the buffers and write your own serialize/deserialize functions. It is a bit simpler if the type and dimensions of the arrays are fixed - something like

```julia
global const buffers = Vector{TYPE}[]

type FooVector
    arr::Vector{TYPE}
end

Base.serialize(s::AbstractSerializer, data::FooVector)
    Serializer.serialize_type(s, typeof(data))
    write(s.io, data.arr)
end

function Base.deserialize(s::AbstractSerializer, t::Type{FooArray})
    buffer = isempty(buffers) ? Vector{TYPE}(SIZE) : pop!(buffers)
    readbytes!(s.io, reinterpret(UInt8, buffer))
    return FooVector(buffer)
end

Your code should wrap and send vectors as `FooVector` objects.
Also either install a finalizer that will `push!` back `FooVector.arr` to `buffers` when done or do it manually.

```

For a more generic implementation (any type/shape of bitstype arrays) you should serialize type/shape/size information and handle it appropriately. Functions for serializing and deserializing arrays in `base/serialize.jl` will give you an idea of various use cases.

---

<div class="post-metadata">

**Author:** ![amit.murthy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amit.murthy/32/47_2.png) [@amit.murthy](https://discourse.julialang.org/u/amit.murthy)\
**Post date:** [April 17, 2017, 11:40am UTC](https://discourse.julialang.org/t/parallel-fetch-into-pre-allocated-memory/3238/3 "2017-04-17T11:40:30Z")

</div>

Github issue created - [https://github.com/JuliaLang/julia/issues/21413](https://github.com/JuliaLang/julia/issues/21413)