How to deal with multiple processes trying to load the same file (and crash)?

Specific Domains Data
1

hey folks,

Right now I have the issue that multiple jobs on a cluster try to load the same data (my data is saved using JLD) and there is a problem with the disk, as a result the jobs fail. How can I deal with this issue? The jobs are a series of scientific calculations which save intermediate results, which again can be reused by other jobs. Sometimes these collisions cause the jobs to crash. Is there maybe already a package for that? I imagine something like a database server, where the jobs fetch their data simultaneously (however, note that I do not have admin rights on the cluster).

Best,

v.

2

I’d probably try to use a proper database like SQLite.jl or MySQL.jl (MySQL.jl is only a client, you’d have to manage/start the server as a separate process).

1 Like
3

readonly shouldn’t be a issue right? this is pretty common is research where you have some read-only files hosted in shared filesystem.

4

Exactly, read-only is sufficient