# How to handle and store large amounts of (distributed) generated data?

**URL:** <https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231>\
**Category:** Data\
**Created:** [April 15, 2017, 10:14am UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231 "2017-04-15T10:14:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 15, 2017, 10:14am UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231/1 "2017-04-15T10:14:45Z")

</div>

A lot of (Monte Carlo) simulations can be done simultaneously and independently on many nodes of an HPC, each generating large solutions for later analysis. However, a straight `pmap` will try to build a giant output that likely won’t fit on any node, and then just crash the system.

Are there distributed databases I could instead write information to, or instead somehow serialize the types separately and concatenate them into one big data file?

Related to this post is the other post, discussing what to actually save:

> [@Suggested formats for saving and serialization](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230):
>
> I was wondering what the suggested formats for saving and serializing are these days. I am really confused about the landscape of DataTables, DataFrames, databases, etc. I was hoping to write a few functions for the DiffEq solution type to save to some common data formats, but am not sure what I should be targeting. It should be something that would work well with statistics, plotting, and machine learning libraries. Or maybe the approach is just to go generic: I know there are things like DataS…

and the DiffEq issue monitoring the updates:

> <https://github.com/SciML/DifferentialEquations.jl/issues/151>
>
> There should be ways to easily save out the solution type to data. Something to …save to a DataFrame/DataTable, and something to save to JLD.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [April 15, 2017, 11:21am UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231/2 "2017-04-15T11:21:54Z")

</div>

In the past I have used `DistributedArrays` (you can distribute more than arrays) and the write the data to a series of files on a distributed Filesystem (lustre). In my use cases it was most likely that I would want to read-in the data distributed as well in the end.

There is an mpi extension to HDF5 that might be usable, but I have never used that from within Julia.

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [April 15, 2017, 12:01pm UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231/3 "2017-04-15T12:01:25Z")

</div>

If you know how much data you’ll be writing ahead of time, pre-creating the files and doing pwrites to such files works just fine generally.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 15, 2017, 12:10pm UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231/4 "2017-04-15T12:10:34Z")

</div>

> [@Keno](#):
>
> pre-creating the files and doing pwrites

What’s a pwrite? A quick Google search (SO) says it’s a file writing which uses independent memory and can be done concurrently?

But I don’t see a `pwrite` in Julia, just in the Linux manual. Is there a standard way to do this in Julia?

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [April 15, 2017, 12:15pm UTC](https://discourse.julialang.org/t/how-to-handle-and-store-large-amounts-of-distributed-generated-data/3231/5 "2017-04-15T12:15:39Z")

</div>

Yes, you basically give it an offset and as long as you’re doing stripe aligned writes (writes in multiples of the file system block size), and only write a given strip from one process at a time, it’ll generally be pretty high performance. I just checked whether we have wrapped pwrite in filesystem.jl, but it doesn’t seem like it. For the quick and dirty solution see [https://github.com/jeff-regier/Celeste.jl/blob/master/src/SDSSIO.jl#L736-L782](https://github.com/jeff-regier/Celeste.jl/blob/master/src/SDSSIO.jl#L736-L782), which is the same idea but for reads. At some point we should wrap it in base.