Precompilation error using HPC

zymbuzz · November 2, 2018, 5:58pm

Hi!
I am running a Julia code on university’s HPC and actually it works fine unless I run the same code using two different nodes at the same time. I get this precompilation error, which I don’t how to solve. I guess it has something to do that both processes are using the same Pkg collection, meaning that they are precompiling same modules/packages. Thanks for your help

┌ Warning: Module DataStructures with build ID 9970418894265805 is missing from the cache.
│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module DataStructures with build ID 9970419130506598 is missing from the cache.
│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
│   exception = ArgumentError: Invalid header in cache file /home/userAAA/.julia/compiled/v1.0/PDMats/wuzEE.ji.
└ @ Base loading.jl:963
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
└ @ Base loading.jl:963
ERROR: LoadError: MethodError: no method matching getindex(::ErrorException, ::Int64)
Stacktrace:
 [1] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:615
 [2] _require_from_serialized(::String) at ./loading.jl:678
 [3] _require(::Base.PkgId) at ./logging.jl:317
 [4] require(::Base.PkgId) at ./loading.jl:852
 [5] macro expansion at ./logging.jl:311 [inlined]
 [6] require(::Module, ::Symbol) at ./loading.jl:834
 [7] include at ./boot.jl:317 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1038
 [9] include(::Module, ::String) at ./sysimg.jl:29
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:319 [inlined]
 [12] eval(::Expr) at ./client.jl:389
 [13] top-level scope at ./none:3
in expression starting at /home/userAAA/.julia/packages/TVPVARPkg/yr0dR/src/TVPVARPkg.jl:10
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for TVPVARPkg [da917d6a-ac4c-11e8-3f88-f9a7edfdbf5e]
│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/TVPVARPkg/lg9Wo.ji.
└ @ Base loading.jl:963
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for TVPVARPkg [da917d6a-ac4c-11e8-3f88-f9a7edfdbf5e]
│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/TVPVARPkg/lg9Wo.ji.
└ @ Base loading.jl:963
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for TVPVARPkg [da917d6a-ac4c-11e8-3f88-f9a7edfdbf5e]
│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/TVPVARPkg/lg9Wo.ji.
└ @ Base loading.jl:963
ERROR: LoadError: On worker 2:
Failed to precompile TVPVARPkg [da917d6a-ac4c-11e8-3f88-f9a7edfdbf5e] to /home/userAAA/.julia/compiled/v1.0/TVPVARPkg/lg9Wo.ji.
error at ./error.jl:33
macro expansion at ./logging.jl:313 [inlined]
compilecache at ./loading.jl:1184
_require at ./logging.jl:311
require at ./loading.jl:852
macro expansion at ./logging.jl:311 [inlined]
require at ./loading.jl:834
eval at ./boot.jl:319
#116 at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:276
run_work_thunk at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:56
run_work_thunk at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:65
#102 at ./task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:407
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:398
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:419
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:419
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at ./task.jl:259

...and 4 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:226
 [2] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:207
 [3] top-level scope at /opt/pkgs/update/julia/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:190
 [4] include at ./boot.jl:317 [inlined]
 [5] include_relative(::Module, ::String) at ./loading.jl:1038
 [6] include(::Module, ::String) at ./sysimg.jl:29
 [7] exec_options(::Base.JLOptions) at ./client.jl:229
 [8] _start() at ./client.jl:421
in expression starting at /share/home/userAAA/TVPVAR/runjuliafile24.jl:8
      From worker 20:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
      From worker 20:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/OrderedCollections/LtT3J.ji.
      From worker 20:	└ @ Base loading.jl:963
      From worker 22:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 22:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 22:	└ @ Base loading.jl:963
      From worker 7:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 7:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/DataStructures/xKiwJ.ji.
      From worker 7:	└ @ Base loading.jl:963
      From worker 8:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 8:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 8:	└ @ Base loading.jl:963
      From worker 5:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 5:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 5:	└ @ Base loading.jl:963
      From worker 17:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 17:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 17:	└ @ Base loading.jl:963
      From worker 24:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 24:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 24:	└ @ Base loading.jl:963
      From worker 6:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 6:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 6:	└ @ Base loading.jl:963
      From worker 2:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 2:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 2:	└ @ Base loading.jl:963
      From worker 23:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 23:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/DataStructures/xKiwJ.ji.
      From worker 23:	└ @ Base loading.jl:963
      From worker 21:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 21:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 21:	└ @ Base loading.jl:963
      From worker 16:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 16:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 16:	└ @ Base loading.jl:963
      From worker 3:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 3:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 3:	└ @ Base loading.jl:963
      From worker 14:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 14:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 14:	└ @ Base loading.jl:963
      From worker 18:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 18:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 18:	└ @ Base loading.jl:963
      From worker 9:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 9:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 9:	└ @ Base loading.jl:963
      From worker 11:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 11:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 11:	└ @ Base loading.jl:963
      From worker 15:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 15:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 15:	└ @ Base loading.jl:963
      From worker 10:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 10:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 10:	└ @ Base loading.jl:963
      From worker 19:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 19:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 19:	└ @ Base loading.jl:963
      From worker 13:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 13:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 13:	└ @ Base loading.jl:963
      From worker 12:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
      From worker 12:	│   exception = Required dependency OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] failed to load from a cache file.
      From worker 12:	└ @ Base loading.jl:963
      From worker 22:	┌ Warning: Module DataStructures with build ID 9970418621322988 is missing from the cache.
      From worker 22:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 22:	└ @ Base loading.jl:941
      From worker 8:	┌ Warning: Module DataStructures with build ID 9970418662337312 is missing from the cache.
      From worker 8:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 8:	└ @ Base loading.jl:941
      From worker 11:	┌ Warning: Module DataStructures with build ID 9970419080070126 is missing from the cache.
      From worker 11:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 11:	└ @ Base loading.jl:941
      From worker 18:	┌ Warning: Module DataStructures with build ID 9970419007174339 is missing from the cache.
      From worker 18:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 18:	└ @ Base loading.jl:941
      From worker 17:	┌ Warning: Module DataStructures with build ID 9970418851608119 is missing from the cache.
      From worker 17:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 17:	└ @ Base loading.jl:941
      From worker 2:	┌ Warning: Module DataStructures with build ID 9970418905781229 is missing from the cache.
      From worker 2:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 2:	└ @ Base loading.jl:941
      From worker 23:	┌ Warning: Module DataStructures with build ID 9970418929452802 is missing from the cache.
      From worker 23:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 23:	└ @ Base loading.jl:941
      From worker 5:	┌ Warning: Module DataStructures with build ID 9970418823865619 is missing from the cache.
      From worker 5:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 5:	└ @ Base loading.jl:941
      From worker 7:	┌ Warning: Module DataStructures with build ID 9970418628162347 is missing from the cache.
      From worker 7:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 7:	└ @ Base loading.jl:941
      From worker 19:	┌ Warning: Module DataStructures with build ID 9970419130506598 is missing from the cache.
      From worker 19:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 19:	└ @ Base loading.jl:941
      From worker 24:	┌ Warning: Module DataStructures with build ID 9970418868039756 is missing from the cache.
      From worker 24:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 24:	└ @ Base loading.jl:941
      From worker 14:	┌ Warning: Module DataStructures with build ID 9970418998683742 is missing from the cache.
      From worker 14:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 14:	└ @ Base loading.jl:941
      From worker 6:	┌ Warning: Module DataStructures with build ID 9970418880434497 is missing from the cache.
      From worker 6:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 6:	└ @ Base loading.jl:941
      From worker 16:	┌ Warning: Module DataStructures with build ID 9970418939477061 is missing from the cache.
      From worker 16:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 16:	└ @ Base loading.jl:941
      From worker 21:	┌ Warning: Module DataStructures with build ID 9970418929829611 is missing from the cache.
      From worker 21:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 21:	└ @ Base loading.jl:941
      From worker 9:	┌ Warning: Module DataStructures with build ID 9970419055074413 is missing from the cache.
      From worker 9:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 9:	└ @ Base loading.jl:941
      From worker 3:	┌ Warning: Module DataStructures with build ID 9970418981895496 is missing from the cache.
      From worker 3:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 3:	└ @ Base loading.jl:941
      From worker 10:	┌ Warning: Module DataStructures with build ID 9970419124250537 is missing from the cache.
      From worker 10:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 10:	└ @ Base loading.jl:941
      From worker 13:	┌ Warning: Module DataStructures with build ID 9970419228495416 is missing from the cache.
      From worker 13:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 13:	└ @ Base loading.jl:941
      From worker 15:	┌ Warning: Module DataStructures with build ID 9970419113775101 is missing from the cache.
      From worker 15:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 15:	└ @ Base loading.jl:941
      From worker 12:	┌ Warning: Module DataStructures with build ID 9970419237442372 is missing from the cache.
      From worker 12:	│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
      From worker 12:	└ @ Base loading.jl:941
      From worker 20:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 20:	│   exception = Required dependency SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c] failed to load from a cache file.
      From worker 20:	└ @ Base loading.jl:963
      From worker 4:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 4:	│   exception = Required dependency SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c] failed to load from a cache file.
      From worker 4:	└ @ Base loading.jl:963
      From worker 22:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 22:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 22:	└ @ Base loading.jl:963
      From worker 8:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 8:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 8:	└ @ Base loading.jl:963
      From worker 11:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 11:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 11:	└ @ Base loading.jl:963
      From worker 6:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 6:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 6:	└ @ Base loading.jl:963
      From worker 19:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 19:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 19:	└ @ Base loading.jl:963
      From worker 14:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 14:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 14:	└ @ Base loading.jl:963
      From worker 3:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 3:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/StatsBase/EZjIG.ji.
      From worker 3:	└ @ Base loading.jl:963
      From worker 18:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 18:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 18:	└ @ Base loading.jl:963
      From worker 2:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 2:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 2:	└ @ Base loading.jl:963
      From worker 23:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 23:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 23:	└ @ Base loading.jl:963
      From worker 7:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 7:	│   exception = ArgumentError: Invalid header in cache file /home/userAAA/.julia/compiled/v1.0/PDMats/wuzEE.ji.
      From worker 7:	└ @ Base loading.jl:963
      From worker 17:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 17:	│   exception = ArgumentError: Invalid header in cache file /home/userAAA/.julia/compiled/v1.0/PDMats/wuzEE.ji.
      From worker 17:	└ @ Base loading.jl:963
      From worker 24:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 24:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 24:	└ @ Base loading.jl:963
      From worker 5:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 5:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 5:	└ @ Base loading.jl:963
      From worker 16:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 16:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 16:	└ @ Base loading.jl:963
      From worker 9:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 9:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 9:	└ @ Base loading.jl:963
      From worker 21:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 21:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 21:	└ @ Base loading.jl:963
      From worker 10:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 10:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 10:	└ @ Base loading.jl:963
      From worker 15:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PDMats [90014a1f-27ba-587c-ab20-58faa44d9150]
      From worker 15:	│   exception = Required dependency Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97] failed to load from a cache file.
      From worker 15:	└ @ Base loading.jl:963
      From worker 4:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b]
      From worker 4:	│   exception = ArgumentError: Invalid checksum in cache file /home/userAAA/.julia/compiled/v1.0/SpecialFunctions/78gOt.ji.
      From worker 4:	└ @ Base loading.jl:963
      From worker 13:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa]
      From worker 13:	│   exception = ArgumentError: Invalid header in cache file /home/userAAA/.julia/compiled/v1.0/Rmath/0iTqX.ji.
      From worker 13:	└ @ Base loading.jl:963
      From worker 24:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 24:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 24:	└ @ Base loading.jl:963
      From worker 20:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 20:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 20:	└ @ Base loading.jl:963
      From worker 2:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 2:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 2:	└ @ Base loading.jl:963
      From worker 16:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 16:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 16:	└ @ Base loading.jl:963
      From worker 12:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 12:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 12:	└ @ Base loading.jl:963
      From worker 11:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 11:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 11:	└ @ Base loading.jl:963
      From worker 18:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 18:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 18:	└ @ Base loading.jl:963
      From worker 7:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 7:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 7:	└ @ Base loading.jl:963
      From worker 23:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]
      From worker 23:	│   exception = Required dependency Rmath [79098fc4-a85e-5d69-aa6a-4863f24498fa] failed to load from a cache file.
      From worker 23:	└ @ Base loading.jl:963
      From worker 8:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c]

 
and so on

kristoffer.carlsson · November 2, 2018, 6:06pm

This particular error is fixed with https://github.com/JuliaLang/julia/pull/29325 and will be in 1.0.2. Unfortunately, it just means that another more descriptive error message will be thrown.

JaredCrean2 · November 2, 2018, 6:49pm

There are two approaches:

Run julia with --compilecache=no, which will disable precompilation. This may adversely effect load times (although for the packages I have benchmarked the effect was not that large)
Make sure the precompiled files exist before you launch the job, so each process only has to read the precompile files, not write to them. If you know all the packages you will be using, you can run a little script that does julia -e 'using pkgname' for all the packages. This script should only have to be run once after you update the packages.

(note: I use approach 2, but have not attempted approach 1)

ksmcreynolds · November 2, 2018, 7:18pm

I also ran into this problem and solved it by using a local cache for each job.

# make a local temp directory for the cache
TMPDIR=`mktemp -d`
mkdir "$TMPDIR/compiled"
# copy the cache to the temp directory
rsync -au "$JULIA_DEPOT_PATH/compiled/v1.0" "$TMPDIR/compiled/" 
# set the temp directory as the first depot so any (re)compilation will happen there and not interfere with other jobs
export JULIA_DEPOT_PATH="$TMPDIR:$JULIA_DEPOT_PATH"

If you don’t have $JULIA_DEPOT_PATH set then you need to substitute the default which is $HOME/.julia.

However, I’m not sure if the extra complexity is worth it vs just running with --compilecache=no.

rkat · November 11, 2018, 5:06am

This does the trick. Here is the above transliterated into Julia:

d = strip(String(read(`mktemp -d`)))
mkdir(joinpath(d, "compiled"))
run(`rsync -au $(DEPOT_PATH[1])/compiled $d/compiled`)
pushfirst!(DEPOT_PATH, d)

I found that simply doing --compiled-modules=no was not enough.

jobjob · November 26, 2018, 3:33pm

@kristoffer.carlsson Do you know if there’s a julialang issue to follow for the root cause here?

The basic triggering condition for me is:

change some code in a devved package
run code on multiple cores/nodes using mpirun (without getting the package precompiled)

Usually, though seemingly not always, making sure everything has been precompiled before doing a distributed run, works. Though sometimes it fails - the error rate seems to be correlated with the number of nodes/processes I’m running.

My stack traces (on each process) look similar to the OP and I also see a pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)

Cheers

tkf · November 26, 2018, 6:15pm

I guess this is due to Base.compilecache in different process writing the cache to the same file? If so, how about tweaking Base.compilecache to:

Output compile cache to a process-specific temporary file in the same file system as cachefile (say $(cachefile).$(gethostname()).$(getpid()) ).
rename the temporary file to cachefile (which is supposed to be atomic … even on a NFS?).

jobjob · November 27, 2018, 8:45am

Yeah that might work, this SO post suggests there may be a subtlety if another process tries to read $cachefile (to check if it needs to be updated) on an NFS while it’s being overwritten by the process local cachefile: linux - rename() atomicity and NFS? - Stack Overflow. It may not actually be a serious problem, just might cause an unneeded re-compilation, but I don’t fully grok all the details.

tkf · November 27, 2018, 9:03pm

If re-compilation creates an identical cache file (or close enough file?), maybe that’s fine?

tkf · November 28, 2018, 7:39am

I made a PR Make compilecache atomic by tkf · Pull Request #30174 · JuliaLang/julia · GitHub

tkf · November 30, 2018, 3:37am

ksmcreynolds:

I also ran into this problem and solved it by using a local cache for each job.

# make a local temp directory for the cache
TMPDIR=`mktemp -d`
mkdir "$TMPDIR/compiled"
# copy the cache to the temp directory
rsync -au "$JULIA_DEPOT_PATH/compiled/v1.0" "$TMPDIR/compiled/" 
# set the temp directory as the first depot so any (re)compilation will happen there and not interfere with other jobs
export JULIA_DEPOT_PATH="$TMPDIR:$JULIA_DEPOT_PATH"

By the way, do you need rsync there? I thought julia’s depot system was designed such that cache files in non-primary depot can be used if they are not stale.

ksmcreynolds · November 30, 2018, 3:57pm

@tkf you are correct. julia’s depot system is smarter that I give it credit for.

tkf · November 30, 2018, 11:28pm

Thanks for checking it out!

jacob-roth · March 27, 2019, 9:13pm

If I have a Project.toml and Manifest.toml, is there a better way to avoid the using ... script? I’ve tried instantiate-ing, but this doesn’t seem to work?

Xing_Shi_Cai · May 10, 2019, 9:05am

Approach 2 solves my problem!!

simonbyrne · June 19, 2019, 7:23pm

I’ve written a small package to help with this task:

Feedback appreciated!

johnh · June 20, 2019, 5:31am

@simonbyrne That sounds an excellent idea. You refer to a shared filesystem on an HPC cluster, which entirely fine. I Would like to expand on this a little. On an HPC cluster you may have a home directory which is NFS mounted on all compute nodes. It is generally a bad idea to use your home directory to store executable code and libraries (though see below). When a job starts there will be multiple requests for those files leading to bad performance or maybe timeouts.
You are encouraged to put files like that onto the fast or scratch storage on a cluster - and indeed the Julia package management system should make this easy.

I do have to say that home directories may be on the fast/scratch storage, in which case my point does not apply.
I have seen sites where the quota in the home directory is tiny - precisely to encourage you to store only login scripts and keys there and to discourage ‘running from your home directory’.
Consult your local systems administrators. Bring them cookies.

simonbyrne · June 20, 2019, 3:44pm

That’s a good point: you can set the JULIA_DEPOT_PATH directory which should help in that regard.

JaredCrean2 · June 20, 2019, 5:10pm

I’m not familiar with the new package manager, so I have a couple questions:

On line 8, why DEPOT_PATH[1]? Can there be multiple entries in DEPOT_PATH?
What is the machinery behind Pkg.API.precompile()? In particular, what happens if a package has submodules whose names are exported but are never used or imported in the package. Does precompile still pick up on them? If not, it might be good to have a way for packages to register additional using statements to trigger precompilation of submodules.

As for comments, I would get rid of the catch clause. It seems to make a few assumptions which may not be valid:

a package not being built is the only reason precompile might fail
The current environment is configured for compiling code

The first one might not be true because of, eg. disk quota issue, misconfigured environment variables. The second one might not be true because the right versions of compilers might not be loaded

Also, the Readme suggests using PkgLock from the job submission script, but I would recommend using it immediately after updating packages, so it only has to be done once, rather than for every job (I recall a github issue somewhere that reported precompiling a package and then using the package is slower than using the package with no precompiliation, so for PkgLock to be beneficial, it has to be faster than disabling precompilation).

simonbyrne · June 20, 2019, 8:58pm

yes, to support various level depots (user, local, system, etc)

I’m not sure exactly, but I think precompilation is done at the package level, not the module level.