Accumulating distributed data

grahamas
Wed, 28 Aug 2019 15:56:51 +0000
Data

I’m running distributed simulations and hoping to save the results as I go – the aggregate results are too big to hold in memory. Can someone point me to a good way to do this that doesn’t amount to saving each run in a separate file? JuliaDB almost seems right, but the functionality for out-of-core processing seems to assume all the data already exists. So the closest I can come is saving each batch as a CSV (!) and then using loadtable to re-process the directory into a distributable binary directory.

1 post - 1 participant