# Pmap: does it copy or share arguments across processes?

**URL:** <https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317>\
**Category:** Julia at Scale\
**Tags:** parallel\
**Created:** [November 26, 2017, 6:14am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317 "2017-11-26T06:14:52Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![drjoke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drjoke/32/23562_2.png) [@drjoke](https://discourse.julialang.org/u/drjoke)\
**Post date:** [November 26, 2017, 6:14am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/1 "2017-11-26T06:14:52Z")

</div>

```julia
for (name, security) in dataset.securities
    push!(arg_array, (name, data[name]))
end
tuples = pmap(extract_variables, arg_array)

```

In the above code, data[name]) is a Dict{Int64, Float32} that contains a lot of data.

This document [https://docs.julialang.org/en/stable/manual/parallel-computing](https://docs.julialang.org/en/stable/manual/parallel-computing) is a very hard read, and it makes me think that arg\_array will always be copied to a new process that is running extract\_variables unless I resort to something such as Shared Arrays.

If that is true, how can I make data[name] be shared across all processes in pmap so that it will not be copied and cost some efficiency.

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [November 26, 2017, 6:25am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/2 "2017-11-26T06:25:57Z")

</div>

Yes, you read it correctly, data is not shared for the `pmap` or `@parallel` for loops. Also you did find one of the answers: SharedArrays. The other is DistributedArrays.jl. Last, you can try and use `Threads.@treads`, which shares data.

---

<div class="post-metadata">

**Author:** ![drjoke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drjoke/32/23562_2.png) [@drjoke](https://discourse.julialang.org/u/drjoke)\
**Post date:** [November 26, 2017, 6:55am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/3 "2017-11-26T06:55:06Z")

</div>

Thank you. No wonder why in all my benchmarking, pmap is not faster than map.

---

<div class="post-metadata">

**Author:** ![drjoke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drjoke/32/23562_2.png) [@drjoke](https://discourse.julialang.org/u/drjoke)\
**Post date:** [November 26, 2017, 7:00am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/4 "2017-11-26T07:00:21Z")

</div>

And unfortunately neither SharedArrays nor DistributedArrays.jl will work with Dict.

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [November 26, 2017, 7:31am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/5 "2017-11-26T07:31:14Z")

</div>

DistributedArrays can now hold any type of data with Julia\>v0.6.

---

<div class="post-metadata">

**Author:** ![drjoke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drjoke/32/23562_2.png) [@drjoke](https://discourse.julialang.org/u/drjoke)\
**Post date:** [November 26, 2017, 8:35am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/6 "2017-11-26T08:35:18Z")

</div>

After reading the documentation, it appears that Threads.@threads is what I am looking for as it does not spawn a new process and can share memory.

```julia
len = length(arg_array)
secs = Array{Data.Security, 1}(len)
Threads.@threads for i = 1:len
    secs[i] = read_csv_and_init(arg_array[i])
end

```

I am getting Bus error: 10 from the code above.

It is interesting because without Threads.@threads this code runs fine.

read\_csv\_and\_init does not cause any side effect either as it does not modify any of its input arguments nor access global variables.

I am on Mac OS High Sierra.

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [November 26, 2017, 9:15am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/7 "2017-11-26T09:15:26Z")

</div>

Reading files in parallel is probably not the best. Usually that is limited by disc access not CPU. I don’t know whether that could cause the error too.

---

<div class="post-metadata">

**Author:** ![drjoke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drjoke/32/23562_2.png) [@drjoke](https://discourse.julialang.org/u/drjoke)\
**Post date:** [November 26, 2017, 9:36am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/8 "2017-11-26T09:36:25Z")

</div>

This code gets Bus error: 10

```julia
function test(filename)
    readdlm(string("data/yahoo/", filename), ',')
end

Threads.@threads for filename = readdir("data/yahoo/")
    test(filename)
end

```

While this code works fine.

```julia
function test(filename)
    readdlm(string("data/yahoo/", "2S.csv"), ',')
end

Threads.@threads for filename in readdir("data/yahoo/")
    test(filename)
end

```

It appears that the error is caused when trying to access filename from readdlm where as the string literal “2S.csv” works fine.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 26, 2017, 9:52am UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/9 "2017-11-26T09:52:56Z")

</div>

I don’t believe IO is thread-safe yet (ever?). Printing in threads will do the same.

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [November 26, 2017, 4:34pm UTC](https://discourse.julialang.org/t/pmap-does-it-copy-or-share-arguments-across-processes/7317/10 "2017-11-26T16:34:02Z")

</div>

For printing `Core.println` works.