# Accumulating distributed data

**URL:** <https://discourse.julialang.org/t/accumulating-distributed-data/28117>\
**Category:** Data\
**Tags:** juliadb\
**Created:** [August 28, 2019, 3:56pm UTC](https://discourse.julialang.org/t/accumulating-distributed-data/28117 "2019-08-28T15:56:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Post date:** [August 28, 2019, 3:56pm UTC](https://discourse.julialang.org/t/accumulating-distributed-data/28117/1 "2019-08-28T15:56:52Z")

</div>

I’m running distributed simulations and hoping to save the results as I go – the aggregate results are too big to hold in memory. Can someone point me to a good way to do this that doesn’t amount to saving each run in a separate file? JuliaDB _almost_ seems right, but the functionality for out-of-core processing seems to assume all the data already exists. So the closest I can come is saving each batch as a CSV (!) and then using `loadtable` to re-process the directory into a distributable binary directory.