# How to save parallel outputs to the same file on HPC cluster

**URL:** <https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [April 21, 2020, 10:04pm UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993 "2020-04-21T22:04:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [April 21, 2020, 10:04pm UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/1 "2020-04-21T22:04:45Z")

</div>

I am doing Monte Carlo study on HPC cluster. I parallel the 100 Monte Carlo replications using job array in Slurm. I want to save the results of each replication to the same file. Let’s say for replication ID `n`, I have two results, both of them are small `Vector{Float64}`. Then, I would like to save them as follows:

```julia
n_1 result_1 result_2
n_2 result_1 result_2
n_3 result_1 result_2

```

I think I am looking for a proper package for saving data on HPC cluster and some code like this

```julia
save!("file", "$SLURM_ARRAY_TASK_ID", "$result_1", "$result_2")

```

Any suggestions will be appreciated.

---

<div class="post-metadata">

**Author:** ![xzackli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xzackli/32/38301_2.png) [@xzackli](https://discourse.julialang.org/u/xzackli)\
**Post date:** [April 22, 2020, 12:44am UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/2 "2020-04-22T00:44:16Z")

</div>

One way to do this would be to use MPI.jl with HDF5.jl. You will need to point it at an existing HDF5 installation on your cluster, as parallel HDF5 is not enabled in the HDF5\_jll binary. There is a [test](https://github.com/JuliaIO/HDF5.jl/blob/master/test/mpio.jl) in the HDF5 package that may serve as a reference for you. There doesn’t seem to be a lot of documentation on this, but the Python wrapper h5py’s [documentation](http://docs.h5py.org/en/stable/mpi.html) may also help you.

Practically, I usually just write a bunch of separate data files to cluster scratch, and then run another job to combine them if necessary.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [April 22, 2020, 3:06am UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/3 "2020-04-22T03:06:00Z")

</div>

Whenever I can, I use a database to handle concurrency. For example, [LibPQ.jl](https://github.com/invenia/LibPQ.jl) and [SQLite.jl](https://github.com/JuliaDatabases/SQLite.jl) are good. I believe using a database, even an ephemeral one, will outperform manually handling locks to the `IO`.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [April 22, 2020, 6:09am UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/4 "2020-04-22T06:09:46Z")

</div>

IIRC single-file storage systems like HDF5 and SQLite don’t work on NFS. How do people handle this on HPC cluster? (Scratch space?) I just use a dumb solution that allocates different file path for different “task” as it is OK for my usecase.

---

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [April 22, 2020, 7:03am UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/5 "2020-04-22T07:03:07Z")

</div>

Thanks for your reply. So it seems that it is a little bit difficult to do it. If I instead save result of each replication to a unique file, I wonder how I should do it? When running locally, I use the `DelimitedFiles.jl` package. But I couldn’t find where it saves the result on HPC.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 22, 2020, 8:48am UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/6 "2020-04-22T08:48:36Z")

</div>

Actually, you might have a good method there.  
Write to local storage on the node into $TMPDIR and at the end of the job bring all the output files to the final storage place.  
Local storage is most often fastest.

But to write to HDF5 format with MPI-IO you need a filesystem which will support it.  
You say Scratch space - I think your scratch storage probably does support it.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [April 22, 2020, 10:40pm UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993/7 "2020-04-22T22:40:10Z")

</div>

> [@zxjroger](#):
>
> But I couldn’t find where it saves the result on HPC.

As a sanity check, I’d make sure that the working directories of the julia processes are the intended one. If not, maybe just using a full path would fix it?

> [@johnh](#):
>
> Local storage is most often fastest.
> 
> But to write to HDF5 format with MPI-IO you need a filesystem which will support it.

Thanks! Yeah, I think I’ll use those techniques if I have an I/O-intensive program.