# Repeated precompilation on a cluster makes life difficult
**URL:** https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194
**Category:** General Usage
**Tags:** question, hpc, pkg, precompilation
**Created:** [February 7, 2023, 10:56am UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194 "2023-02-07T10:56:43Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 10:56am UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/1 "2023-02-07T10:56:43Z")
I’m using julia v1.9.0-beta3 on a SLURM cluster. On a login node with `versioninfo`
```julia
julia> versioninfo()
Julia Version 1.9.0-beta3
Commit 24204a73447 (2023-01-18 07:20 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 64 × AMD EPYC 7742 64-Core Processor
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, znver2)
Threads: 1 on 64 virtual cores
```
if I run
```julia
OMP_NUM_THREADS=1 julia --project --startup=no -e "import Pkg; Pkg.instantiate(); Pkg.precompile()"
```
this precompiles dependencies in my project (110 overall, so this takes a while). After this, I log in to a compute node with `versioninfo`
```julia
julia> versioninfo()
Julia Version 1.9.0-beta3
Commit 24204a73447 (2023-01-18 07:20 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 40 × Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
Threads: 1 on 40 virtual cores
```
and run
```julia
julia> import Pkg; Pkg.precompile()
Precompiling environment...
110 dependencies successfully precompiled in 541 seconds. 8 already precompiled.
```
This seems to precompile dependencies again, which is quite time-consuming.
Once this is over, though, if I return to the login node, and run
```julia
julia> import Pkg; Pkg.precompile()
Precompiling environment...
110 dependencies successfully precompiled in 443 seconds. 8 already precompiled.
```
this restarts the precompilation process all over again! Why is everything being precompiled repeatedly, when nothing has changed in the dependencies or the source? The cycle continues if I try it on the compute node subsequently.
I have tried deleting the compile cache, which didn’t change anything. Ideally, Julia would need to precompile dependencies only once on each architecture.
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [February 7, 2023, 12:05pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/2 "2023-02-07T12:05:37Z")
Is the path of Julia the same between compute and login node? If not, that’d invalidate the cache. But having to always precompile the environment is definitely not how this is supposed to work. You can try setting `JULIA_DEBUG=loading` to try and understand why the cache is being invalidated.
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 1:50pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/3 "2023-02-07T13:50:25Z")
Thanks for that suggestion! The path of the Julia executable is the same on both nodes, but I find a bunch of messages similar to the following when precompilation restarts:
```julia
┌ Debug: Rejecting cache file /scratch/user/.julia/compiled/v1.9/Combinatorics/AwRuT_oTZDM.ji for [top-level] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2641
```
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 4:08pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/4 "2023-02-07T16:08:37Z")
I see the same issue on a freshly installed nightly as well
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [February 7, 2023, 5:07pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/6 "2023-02-07T17:07:57Z")
On login node
CPU: 64 × AMD EPYC 7742 64-Core Processor
and on the compute node
CPU: 40 × Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
An expert will be along in a minute to enlighten us.
Looks to me like the compiler targets are different
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 5:10pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/7 "2023-02-07T17:10:01Z")
I’ve received some help on [the issue](https://github.com/JuliaLang/julia/issues/48579) that I’ve created about this, but it does seem to be related to different compiler targets
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [February 7, 2023, 5:17pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/8 "2023-02-07T17:17:37Z")
This is an interesting problem. In general in HPC you would build/precompile an optimised binary for the target architecture you intend to run it on.
In the old days we would architect a system where login/compile nodes have the same CPU families.
These days with things changing so fast, and also virtualised / containerised servers you can’t rely on that rather naive assumption.
What is the relevant JULIA\_CPU\_TARGET for your Xeon CPU?
Googling - I saw a reference to archspec from the Spack folks (I heard Todd Gamblin talk at FOSDEM on Friday)
Off topic question - does Archspec help us choose JULIA\_CPU\_TARGET ?
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 5:22pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/9 "2023-02-07T17:22:14Z")
Unfortunately, I don’t know enough about CPU targets. How does one find this out?
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [February 7, 2023, 5:45pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/10 "2023-02-07T17:45:29Z")
Did it also happen on Julia 1.8? The Pluto package has/had it too. I guessed it was caused by one of the dependencies but never figured it out.
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 5:56pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/11 "2023-02-07T17:56:13Z")
This does not happen on v1.8, but happens on v1.9 onwards. Perhaps this is related to native code caching?
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 7, 2023, 6:20pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/12 "2023-02-07T18:20:44Z")
>
>
> it is common for a user of some cluster to create remote processes via e.g. http…s://github.com/JuliaParallel/ClusterManagers.jl, the workflow essentially looks like this:
> \`\`\`julia
> | | |\_| | | | (\_| | | Version 1.8.5 (2023-01-08)
> \_/ |\\\_\_'\_|\_|\_|\\\_\_'\_| | Official https://julialang.org/ release
> |\_\_/ |
>
> julia\> using WVZAnalysis, ClusterManagers, Distributed
>
> julia\> addprocs(HTCManager(80))
> Waiting for 80 workers: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 .
>
> julia\> @everywhere using WVZAnalysis
> \`\`\`
>
> this has stopped working on 1.9-beta2 with errors like:
> \`\`\`julia
> julia\> @everywhere using WVZAnalysis
> From worker 2: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SentinelArrays \[91c51154-3ec4-41a3-a24f-3f23e20d615c\]
> From worker 2: │ exception = ArgumentError: Invalid checksum in cache file /home/jiling/.julia/compiled/v1.9/SentinelArrays/uMYVe\_zIiTQ.so.
> From worker 2: └ @ Base loading.jl:1673
> From worker 3: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SentinelArrays \[91c51154-3ec4-41a3-a24f-3f23e20d615c\]
> From worker 3: │ exception = ArgumentError: Invalid checksum in cache file /home/jiling/.julia/compiled/v1.9/SentinelArrays/uMYVe\_zIiTQ.so.
> From worker 3: └ @ Base loading.jl:1673
> From worker 8: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for JLLWrappers \[692b3bcd-3c85-4b1f-b108-f13ce0eb3210\]
> From worker 8: │ exception = Required dependency Preferences \[21216c6a-2e73-6563-6e65-726566657250\] failed to load from a cache file.
> From worker 8: └ @ Base loading.jl:1673
> From worker 2: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for JLLWrappers \[692b3bcd-3c85-4b1f-b108-f13ce0eb3210\]
> From worker 2: │ exception = Required dependency Preferences \[21216c6a-2e73-6563-6e65-726566657250\] failed to load from a cache file.
> \`\`\`
>
> might be related to #48057 and #48039
yeah, something is funky with Julia 1.9 and above
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 7, 2023, 6:29pm UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/13 "2023-02-07T18:29:19Z")
Setting the environment variable `export JULIA_CPU_TARGET=generic` before precompiling seems to resolve this issue, as suggested in [Package Images · The Julia Language](https://docs.julialang.org/en/v1.10-dev/devdocs/pkgimg/#Package-images-optimized-for-multiple-microarchitectures)
A better value for this specific issue might be `export JULIA_CPU_TARGET="generic;skylake-avx512,clone_all;znver2,clone_all"`
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [February 8, 2023, 2:57am UTC](https://discourse.julialang.org/t/repeated-precompilation-on-a-cluster-makes-life-difficult/94194/14 "2023-02-08T02:57:27Z")
> [@jishnub](#):
>
> Perhaps this is related to native code caching?
Yes, you compiled native code for an incompatible [Instruction Set Architecture](https://en.m.wikipedia.org/wiki/Instruction_set_architecture) (ISA) and so when you try and load the cache the file is rejected because it’s probably using illegal instructions for your current CPU. By using multi-versioning you circumvent the issue by compiling the code for multiple ISAs.