# 'Stale File Handle' Error When Submitting Job Array on SLURM

**URL:** <https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108>\
**Category:** Julia at Scale\
**Created:** [October 20, 2021, 5:44pm UTC](https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108 "2021-10-20T17:44:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dleather](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dleather/32/43762_2.png) [@dleather](https://discourse.julialang.org/u/dleather)\
**Post date:** [October 20, 2021, 5:44pm UTC](https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108/1 "2021-10-20T17:44:26Z")

</div>

I’m attempting to submit a large number of jobs on an HPC running SLURM. The code runs fine when submitting a single job at a time, but I am submitting 4000 jobs at once. The error is as follows:

> ERROR: SystemError: close: Stale file handle  
> Stacktrace:  
> [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)  
> @ Base ./error.jl:168  
> [2] #systemerror#62  
> @ ./error.jl:167 [inlined]  
> [3] systemerror  
> @ ./error.jl:167 [inlined]  
> [4] close  
> @ ./iostream.jl:63 [inlined]  
> [5] open(::Pkg.API.var"#248#249"{Vector{Pkg.Types.PackageSpec}}, ::String, ::Vararg{String, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})  
> @ Base ./io.jl:332  
> [6] open  
> @ ./io.jl:328 [inlined]  
> [7] save\_precompile\_state()  
> @ Pkg.API /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1279  
> [8] precompile(ctx::Pkg.Types.Context; internal\_call::Bool, strict::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})  
> @ Pkg.API /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1210  
> [9] \_auto\_precompile(ctx::Pkg.Types.Context)  
> @ Pkg /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:596  
> [10] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update\_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow\_build::Bool, allow\_autoprecomp::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})  
> @ Pkg.API /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1357  
> [11] instantiate  
> @ /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1325 [inlined]  
> [12] #instantiate#252  
> @ /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1321 [inlined]  
> [13] instantiate()  
> @ Pkg.API /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1321  
> [14] top-level scope  
> @ none:1

I suspect this is a race condition. Following some other advice found on Discourse, I’ve tried calling the code using the following to no avail:

`~/julia-1.6.2/bin/julia --project=@. -e 'using Pkg; Pkg.instantiate(); Pkg.precompile(); include(PATH_TO_SCRIPT)'`

I’ve tried adding random sleep time before the above code, which has allowed maybe 70% of my jobs to start successfully, but I am looking for a 100% solution as computing resources are costly, and I don’t want to pay for random sleep time.

---

<div class="post-metadata">

**Author:** ![dleather](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dleather/32/43762_2.png) [@dleather](https://discourse.julialang.org/u/dleather)\
**Post date:** [October 20, 2021, 9:20pm UTC](https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108/2 "2021-10-20T21:20:48Z")

</div>

The issue was resolved using @simonbyrne 's PkgLock.jl ([https://github.com/simonbyrne/PkgLock.jl](https://github.com/simonbyrne/PkgLock.jl)).

The submission code was changed to the following after installing PkgLock.jl as a shared environment:

```julia
~/julia-1.6.2/bin/julia --project -e 'push!(LOAD_PATH, "@pkglock"); using PkgLock; PkgLock.instantiate_precompile()'
~/julia-1.6.2/bin/julia --project -e 'include(PATH_TO_SCRIPT)'

```