# DrWatson using tmpsave to save many files fails

**URL:** <https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483>\
**Category:** General Usage\
**Tags:** question\
**Created:** [August 16, 2021, 1:18pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483 "2021-08-16T13:18:52Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 1:18pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/1 "2021-08-16T13:18:52Z")

</div>

I am using DrWatson to run simulations on a cluster. I use this scheme to run many jobs - [Real World Examples · DrWatson](https://juliadynamics.github.io/DrWatson.jl/dev/real_world/#Using-a-Serial-Cluster-1). The problem is that when I use too many parameters (which generate ~180 files), `tmpsave` fails.  
I get this error:

```julia
signal (7): Bus error
in expression starting at /home/labs/orenraz/roiho/Mpemba/AntiFerroMonteCarlo/scripts/monte_carlo_prepare_job.jl:43
unsafe_store! at ./pointer.jl:118 [inlined]
unsafe_store! at ./pointer.jl:118 [inlined]
jlunsafe_store! at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/JLD2.jl:42 [inlined]
jlunsafe_store! at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/misc.jl:15 [inlined]
_write at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/mmapio.jl:190 [inlined]
jlwrite at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/misc.jl:27 [inlined]
commit at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/datatypes.jl:275
h5fieldtype at /home/labs/orenraz/roiho/.julia/packages/JLD2/KN6F6/src/data/writing_datatypes.jl:364
unknown function (ip: 0x7f7e4b771a3d)

```

When I run the same script on my machine, it works fine.  
So probably it is related to the cluster. Does anyone know how to solve this?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 1:57pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/2 "2021-08-16T13:57:37Z")

</div>

How are you calling tmpsave ?  
I guess the directory ‘tmp’ is actually /home/username/tmp  
If tmp is actually /tmp this is different on each cluster node

I would start by looking at your limits when running a job. It would be very unusual to be limited to 180 open files - but who knows.  
Assuming you are using Slurm start an interactive job then show the limits  
srun --nodes=1 --pty bash -i

> ulimit -a  
> exit

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 2:15pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/3 "2021-08-16T14:15:03Z")

</div>

when I call `tmpsave` I simply do `tmpsave(dicts)`, which writes automatically to the folder `_research/tmp/`. This folder is located in the project folder of DrWatson.

I am not sure exactly how to identify the cluster so that people recognize it. Is IBM HPC sufficient?

Just to make sure I was clear: I am running a julia script which save 180 files, and then submits 180 jobs where each submission takes a file as an argument.  
So I encounter the problem before the submission even - simply running the julia script causes the error when `tmpsave` is executed.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 3:02pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/4 "2021-08-16T15:02:06Z")

</div>

@roi.holtzman An IBM HPC probably will be using LSF  
This is an IBM Blue Gene and not a Lenovo system?  
I would still run an interactive batch job and have a look what filesystems are present when you are running under batch, and if you can ‘see’ those tmp files

I tis many years since I used LSF, however I think this should work

bsub -Is bash

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 3:09pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/5 "2021-08-16T15:09:33Z")

</div>

Emmmm… Looking at the pages for WEXAC… are you sure that the home folder on your personal machine is the same as your home folder on the HPC cluster?

I think you might have to do that interactive login and either  
a) sync across those files  
or  
b) run tmpsave actually on the cluster storage

It looks like you are using Spectrum Scale (GPFS) storage on the HPC.  
Can you ssh into the HPC and run these commands:  
pwd  
df -h .

DO you see the aame set of files you see on your personal machine? Probably not…

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 3:13pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/6 "2021-08-16T15:13:39Z")

</div>

Can you ssh into the HPC cluster?

Once we have this figured out it might be a possible to moun the HPC home directory as an sshfs filesystem.  
You can then drag and drop files between both locations.  
I did not tell you this… 😉

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 3:19pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/7 "2021-08-16T15:19:03Z")

</div>

I can ssh to the cluster. This is how I run the jobs.  
You’re right that the files I have on the cluster are not like I have on my own machine. However, I put everything there. All of the files are there.

I think that a crucial hint is that when I have a small amount of files (~20) everything works fine.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 3:24pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/8 "2021-08-16T15:24:10Z")

</div>

This is weird. The ulimits I have on an example HPC cluster are below.  
Try ulimit -a please.

$ ulimit -a  
core file size (blocks, -c) unlimited  
data seg size (kbytes, -d) unlimited  
scheduling priority (-e) 0  
file size (blocks, -f) unlimited  
pending signals (-i) 1028842  
max locked memory (kbytes, -l) unlimited  
max memory size (kbytes, -m) 1048576  
open files (-n) 65536  
pipe size (512 bytes, -p) 8  
POSIX message queues (bytes, -q) 819200  
real-time priority (-r) 0  
stack size (kbytes, -s) 10240  
cpu time (seconds, -t) unlimited  
max user processes (-u) 250556  
virtual memory (kbytes, -v) unlimited  
file locks (-x) unlimited

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 3:28pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/9 "2021-08-16T15:28:49Z")

</div>

This is what I find

```julia
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 771761
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 3000000
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 43200
max user processes (-u) 4096
virtual memory (kbytes, -v) 3000000
file locks (-x) unlimited

```

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 3:34pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/10 "2021-08-16T15:34:29Z")

</div>

> [@roi.holtzman](#):
>
> `open files (-n) 1024`

That seems small to me. I usually have root access so would use the ‘lsof’ command to count the file descriptors a user has open.  
Run ‘lsof -u youriserid’

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 3:43pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/11 "2021-08-16T15:43:46Z")

</div>

Not a lot. The number of lines is 90:

```julia
$ lsof -u roiho | wc -l
90

```

Is the full output important? I didn’t want to spam…

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 16, 2021, 4:33pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/12 "2021-08-16T16:33:37Z")

</div>

The full output is not important - just the number of file descriptors.  
My theory is that when you run with the 180 files you are exceeding the 1024 limit.

I could be very much leading you down the wrong track, so please do not just listen to me!  
Have you asked the team who manage the HPC for some help?

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 16, 2021, 4:57pm UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/13 "2021-08-16T16:57:37Z")

</div>

It does seem like you’re on to something:) But I don’t know how to solve the issue.  
I did contact the HPC team. Hope they will respond soon. Will update if they find a solution.

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 17, 2021, 11:33am UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/14 "2021-08-17T11:33:45Z")

</div>

I have logged in to another access server and managed to get higher limits:

```julia
$ ulimit -a 
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 767357
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 512000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 14400
max user processes (-u) 1024
virtual memory (kbytes, -v) 50194304
file locks (-x) unlimited

```

But unfortunately, this does not solve the issue.  
The cluster teem is trying to figure out what is the issue.

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [August 18, 2021, 11:07am UTC](https://discourse.julialang.org/t/drwatson-using-tmpsave-to-save-many-files-fails/66483/15 "2021-08-18T11:07:21Z")

</div>

The solution is found, and it very simple!  
The space on the cluster has ran out, and therefore the julia script was not able to save the parameter files.  
Deleting files and clearing some space solved the issue.