# Compile errors on HPC

**URL:** <https://discourse.julialang.org/t/compile-errors-on-hpc/47264>\
**Category:** General Usage\
**Tags:** question\
**Created:** [September 25, 2020, 2:09pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264 "2020-09-25T14:09:36Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [September 25, 2020, 2:09pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/1 "2020-09-25T14:09:36Z")

</div>

I am running code via slurm on an HPC. Most of the time, the code compiles without issue. Sometimes, it throws errors such as:

```julia
[lhendri@longleaf-login2 CollegeStrat]$ cat ../log/hProdBounded.out
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Clustering [aaaa29a8-35af-508c-8bc3-b662a17a0fe5]
│ exception = Required dependency NearestNeighbors [b8a86587-4115-5ab1-83bc-aa920d37bbce] failed to load from a cache file.
└ @ Base loading.jl:1041
ERROR: LoadError: SystemError: opening file "/nas/longleaf/home/lhendri/.julia/compiled/v1.4/EconometricsLH/OoVDL_N14SI.ji": Permission denied
Stacktrace:
 [1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
 [2] #systemerror#50 at ./error.jl:167 [inlined]
 [3] systemerror at ./error.jl:167 [inlined]
 [4] open(::String; read::Bool, write::Nothing, create::Nothing, truncate::Nothing, append::Bool) at ./iostream.jl:254
 [5] open(::String, ::String) at ./iostream.jl:310
 [6] open(::Base.var"#692#694", ::String, ::Vararg{String,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:296
 [7] open at ./io.jl:296 [inlined]
 [8] compilecache(::Base.PkgId, ::String) at ./loading.jl:1264
 [9] _require(::Base.PkgId) at ./loading.jl:1029
 [10] require(::Base.PkgId) at ./loading.jl:927
 [11] require(::Module, ::Symbol) at ./loading.jl:922
 [12] include(::Module, ::String) at ./Base.jl:377
 [13] top-level scope at none:2
 [14] eval at ./boot.jl:331 [inlined]
 [15] eval(::Expr) at ./client.jl:449
 [16] top-level scope at ./none:3
in expression starting at /nas/longleaf/home/lhendri/.julia/packages/ModelParams/j9xXf/src/ModelParams.jl:20
ERROR: LoadError: Failed to precompile ModelParams [4089ccbe-b1dc-5f86-a141-4606b18b4241] to /nas/longleaf/home/lhendri/.julia/compiled/v1.4/ModelParams/tRb7n_N14SI.ji.
[...]

```

When I resubmit the same code again, it often runs without issue.  
How can I avoid these kinds of compile errors?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [September 25, 2020, 3:40pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/2 "2020-09-25T15:40:18Z")

</div>

There is a hint that this is a network-attached-storage file system. These are tricky to write to reliably.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 25, 2020, 3:46pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/3 "2020-09-25T15:46:11Z")

</div>

There was a package here which had the fix for race conditions when running on several compute nodes sharing a filesystem… Pkglock

[https://github.com/simonbyrne/PkgLock.jl](https://github.com/simonbyrne/PkgLock.jl)

May I Ask if you are running on multiple compute servers?

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [September 25, 2020, 5:18pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/4 "2020-09-25T17:18:19Z")

</div>

The jobs run on a single node, multiple cores.

You are saying this can create race conditions at compile time?

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [September 25, 2020, 5:19pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/5 "2020-09-25T17:19:58Z")

</div>

If you’re launching multiple jobs at once or using distributed parallelism (either via the Distributed stdlib or MPI), you might be hitting race conditions. Our solution has been to run:

```julia
julia --project -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'

```

before launching any jobs.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [September 25, 2020, 5:42pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/6 "2020-09-25T17:42:15Z")

</div>

Is this with julia 1.5.1? Shouldn’t [https://github.com/JuliaLang/julia/pull/36416](https://github.com/JuliaLang/julia/pull/36416) help here?

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [September 25, 2020, 6:37pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/7 "2020-09-25T18:37:54Z")

</div>

Thank you for those suggestions. Our HPC is still on 1.4 for now.

I will mark @simonbyrne’s answer as the solution while I wait for 1.5.2 to get installed.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 26, 2020, 2:52am UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/8 "2020-09-26T02:52:17Z")

</div>

I would be interested to hear if this solves the problem.  
Also anything about the configuration of your HPC and the type of storage.  
Maybe someone in the future will have a problem with a similar storage setup

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [September 26, 2020, 1:04pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/9 "2020-09-26T13:04:44Z")

</div>

As for the configuration, all the details are at

[https://its.unc.edu/research-computing/techdocs/getting-started-on-longleaf/#System%20Information](https://its.unc.edu/research-computing/techdocs/getting-started-on-longleaf/#System%20Information)

I will experiment and report back. Thanks again.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [October 1, 2020, 2:46pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/10 "2020-10-01T14:46:22Z")

</div>

Reporting back: So far the `precompile` trick has done the job. No more compile errors as the jobs are dispatched.

Thanks again.

---

<div class="post-metadata">

**Author:** ![Dr.Merkwuedigliebe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dr.merkwuedigliebe/32/13187_2.png) [@Dr.Merkwuedigliebe](https://discourse.julialang.org/u/Dr.Merkwuedigliebe)\
**Post date:** [January 2, 2022, 5:04pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/11 "2022-01-02T17:04:31Z")

</div>

I have one related question:  
If one runs the `precompile` trick _while_ existing runs are on-going, does it affect the existing runs?

I have multiple branches (with variant behaviours) of the same module, and the simulations take hours to days to run. I would like to know how best to switch branches, precompile, run job arrays on the current branch, all without impacting the already running jobs.

Many thanks!

---

<div class="post-metadata">

**Author:** ![erny123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erny123/32/52255_2.png) [@erny123](https://discourse.julialang.org/u/erny123)\
**Post date:** [March 11, 2024, 4:55pm UTC](https://discourse.julialang.org/t/compile-errors-on-hpc/47264/12 "2024-03-11T16:55:16Z")

</div>

This seems to still be a problem.

I’m on Julia 1.10.2 and having the same issues as all the rest of the cache race conditions.

Your solution is a bit ambiguous. How is this solving the race conditions?

After this is it safe to run `mpiexec -n 20 julia ./script.jl` ?

In `script.jl` , is it save to do:

```julia
using Pkg
Pkg.activate(".")
using MPI
using HDF5
using PencilArrays
.
.
.

```