I’m running distributed simulations and hoping to save the results as I go – the aggregate results are too big to hold in memory. Can someone point me to a good way to do this that doesn’t amount to saving each run in a separate file? JuliaDB almost seems right, but the functionality for out-of-core processing seems to assume all the data already exists. So the closest I can come is saving each batch as a CSV (!) and then using loadtable
to re-process the directory into a distributable binary directory.