# Adding data to worker processes via @everywhere
**URL:** https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410
**Category:** General Usage
**Created:** [January 9, 2019, 3:48am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410 "2019-01-09T03:48:38Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 9, 2019, 3:48am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/1 "2019-01-09T03:48:38Z")
Hi all,
I know next to nothing about parallel processing, so apologies if this question is obvious. I start julia with `julia -p 2`. Here is a broken simplified version of what my code does:
```
d = read(some_filepath) # d is really big and takes up most of my RAM
some_vector = read(some_other_filepath)
@everywhere n = 1
while n <= N
@everywhere current_d_small = some_function_available_everywhere(d, n)
@everywhere anonymous_function = (x -> another_function_available_everywhere(x, current_d_small)
y = pmap(anonymous_function, some_vector)
n += 1
end
```
This code obviously doesn’t work, because of the line:
```
@everywhere current_d_small = some_function_available_everywhere(d, n)
```
This line errors because `d` is not available everywhere. But I can’t make `d` available everywhere, since a single copy of it takes up most of my RAM.
For the actual work done in `pmap` I only need `current_d_small` to be available to all workers, which is _much_ smaller than `d`. I’ve read the docs, but can’t seem to work out how, on each outer iteration, to get a copy of `current_d_small` to each of the workers so I can make the `pmap` call.
Any help would be much appreciated. Also, bear in mind, I know very little about parallel processing, so maybe I should be using something completely different to `pmap` here.
Thanks,
Colin
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [January 9, 2019, 10:26am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/2 "2019-01-09T10:26:40Z")
If all workers are located on the same machine, you might want to consider threading instead of the functionality in `Distributed`. Threading uses shared memory and is in general more efficient due to less overhead, but also slighly trickier since you must ensure that all operations are thread safe. For example, IO is not currently thread safe. See the [`@threads` macro](https://docs.julialang.org/en/v1/base/multi-threading/index.html#Base.Threads.@threads) to parallelize a for-loop. Also see [the manual on threading](https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Threading-(Experimental)-1).
Your example code does not make sense since you are not using `y` anywhere, but using threads might look something like
```julia
Threads.@threads for n = 1:N
current_d_small = some_function_available_everywhere(d, n)
anonymous_function = (x -> another_function_available_everywhere(x, current_d_small)
y = map(anonymous_function, some_vector) # What to do with y?
end
```
You also have a threaded `map` in [ThreadedMap.jl](https://github.com/GenaBitu/ThreadedMap.jl) and [KissThreading.jl](https://github.com/bkamins/KissThreading.jl)
---
### Author:  [@zgornel](https://discourse.julialang.org/u/zgornel)
#### Post date: [January 9, 2019, 2:10pm UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/3 "2019-01-09T14:10:44Z")
You could also check [SharedArrays](https://docs.julialang.org/en/v1/manual/parallel-computing/#man-shared-arrays-1). Your function `some_function_available_everywhere` as sell as `another_function` should be able to work on chunks of data so there will be manual work involved …
---
### Author:  [@andrewsteck](https://discourse.julialang.org/u/andrewsteck)
#### Post date: [January 9, 2019, 4:21pm UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/4 "2019-01-09T16:21:30Z")
If the above approaches don’t work for you, I’d also recommend checking out [ParallelDataTransfer.jl](https://github.com/ChrisRackauckas/ParallelDataTransfer.jl).
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [January 9, 2019, 4:33pm UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/5 "2019-01-09T16:33:29Z")
Try `@everywhere current_d_small = some_function_available_everywhere($d, n)`
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 10, 2019, 1:46am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/6 "2019-01-10T01:46:37Z")
Thanks for responding. I’ll look into threading. I’d deliberately avoided it up to this point because it was still considered experimental. It sounds like this is only the case for IO now, which doesn’t apply to my current situation.
Regarding `y`, I store it in a pre-allocated array defined outside the outer loop. I omitted that part for simplicity, but perhaps it affects the ability to apply `@threads`?
Thanks again.
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 10, 2019, 1:47am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/7 "2019-01-10T01:47:38Z")
I was already starting to wonder if I should learn about these. Thanks for confirming that I need to 🙂
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 10, 2019, 1:50am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/8 "2019-01-10T01:50:46Z")
Oh wow that seems to do exactly what I was asking for! Many thanks. Chris’s contributions pop up everywhere!
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 10, 2019, 2:01am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/9 "2019-01-10T02:01:21Z")
Thanks for responding. I’m afraid this doesn’t work since it just results in Julia checking for an interpolation of `d` on the other workers (and it isn’t there).
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 10, 2019, 6:27am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/10 "2019-01-10T06:27:50Z")
Unfortunately the loop I want to do in parallel includes a _lot_ of random number generation at different points. The docs suggest a way around this when using `@threads`, but it doesn’t look that fun for a newbie to working in parallel. I might try one of the other solutions. Thanks again.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [January 10, 2019, 6:33am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/11 "2019-01-10T06:33:05Z")
> [@colintbowers](#):
>
> Regarding `y` , I store it in a pre-allocated array defined outside the outer loop. I omitted that part for simplicity, but perhaps it affects the ability to apply `@threads` ?
Unless you access the same part of the array simultaneously from different threads, there should be no problems with this pattern.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [January 10, 2019, 6:35am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/12 "2019-01-10T06:35:03Z")
ParallelDataTransfer.jl is great, but beware that it does copy the sent variable to the memory of each worker, which could be a problem if I understand OP.
---
### Author:  [@colintbowers](https://discourse.julialang.org/u/colintbowers)
#### Post date: [January 11, 2019, 2:53am UTC](https://discourse.julialang.org/t/adding-data-to-worker-processes-via-everywhere/19410/13 "2019-01-11T02:53:33Z")
No, I think this will work fine, as long as I can send `current_d_small` without having to also send `d`. I only need `current_d_small` for the parallel routine and it is _much_ smaller than `d`. Am looking into all the various options now 🙂