# Problem with using Distributed for a Monte Carlo simulation

**URL:** <https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508>\
**Category:** General Usage\
**Tags:** package, parallel, distributed\
**Created:** [March 3, 2023, 4:32pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508 "2023-03-03T16:32:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![greivin](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@greivin](https://discourse.julialang.org/u/greivin)\
**Post date:** [March 3, 2023, 4:32pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/1 "2023-03-03T16:32:40Z")

</div>

I’m using the Distributed package to parallelize a for loop during a Monte Carlo simulation, this for loop is running over several initial conditions to generate a CSV file of L/2 x N entries where L is the size of the system I’m considering and N is the number of different initial conditions.  
for L = 64 and N = 40 the code works perfectly well, however, as soon as I increase the L (for example to L = 128) it throws an error for each worker “Worker # terminated. Unhandled Task ERROR: EOFError: read end of file” or "Worker # terminated. Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET) " that for every worker until a final error “LoadError: TaskFailedException nested task error: ProcessExitedException(2)” is displayed.

I really have no idea what could be going wrong, it seems to be a memory overflow but in principle the clusters in which I’m working on should handle the quantities I’m inserting (on average they have 32 Gb of RAM and 2200-4500 cpu GHz)

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 3, 2023, 6:08pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/2 "2023-03-03T18:08:33Z")

</div>

In my experience, those kinds of error messages (without an explicit segfault stacktrace from the dead worker process) are indeed due to OOM.

---

<div class="post-metadata">

**Author:** ![greivin](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@greivin](https://discourse.julialang.org/u/greivin)\
**Post date:** [March 8, 2023, 4:40pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/3 "2023-03-08T16:40:03Z")

</div>

at the end it seems that an array of ones is consuming a lot of memory, and it is an array of ones of 1048576 times 1048576 that is huge, but at the end every entry in it are just 1s, is there a way of saving memory with an alternative to saving this array explicitly?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 8, 2023, 4:53pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/4 "2023-03-08T16:53:13Z")

</div>

yeah, why do you have a giant array of 1s?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 8, 2023, 5:02pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/5 "2023-03-08T17:02:24Z")

</div>

I think [FillArrays.jl](https://github.com/JuliaArrays/FillArrays.jl) does this?

---

<div class="post-metadata">

**Author:** ![greivin](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@greivin](https://discourse.julialang.org/u/greivin)\
**Post date:** [March 8, 2023, 5:13pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/6 "2023-03-08T17:13:31Z")

</div>

However when I use Ones (from FillArrays) I get that the max size it can get is 100

using FillArrays  
a = Ones(1024, 1024)  
size(a, 1) outputs 100

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 11, 2023, 4:07pm UTC](https://discourse.julialang.org/t/problem-with-using-distributed-for-a-monte-carlo-simulation/95508/7 "2023-03-11T16:07:53Z")

</div>

> [@greivin](#):
>
> a = Ones(1024, 1024)

Hmmm, I don’t see the same:

```julia
julia> using FillArrays

julia> a = Ones(1024, 1024)
1024×1024 Ones{Float64}

julia> a[1]
1.0

julia> a[1, 1]
1.0

julia> a[1, :]
1024-element Ones{Float64}

julia> size(a, 1)
1024

```