Repeated precompilation on a cluster makes life difficult

jling · February 7, 2023, 6:20pm

`compilecache` failed when `@everywhere using` from remote machines

opened 11:24PM - 10 Jan 23 UTC

parallel regression 1.9

it is common for a user of some cluster to create remote processes via e.g. http…s://github.com/JuliaParallel/ClusterManagers.jl, the workflow essentially looks like this: ```julia | | |_| | | | (_| | | Version 1.8.5 (2023-01-08) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> using WVZAnalysis, ClusterManagers, Distributed julia> addprocs(HTCManager(80)) Waiting for 80 workers: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 . julia> @everywhere using WVZAnalysis ``` this has stopped working on 1.9-beta2 with errors like: ```julia julia> @everywhere using WVZAnalysis From worker 2: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SentinelArrays [91c51154-3ec4-41a3-a24f-3f23e20d615c] From worker 2: │ exception = ArgumentError: Invalid checksum in cache file /home/jiling/.julia/compiled/v1.9/SentinelArrays/uMYVe_zIiTQ.so. From worker 2: └ @ Base loading.jl:1673 From worker 3: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SentinelArrays [91c51154-3ec4-41a3-a24f-3f23e20d615c] From worker 3: │ exception = ArgumentError: Invalid checksum in cache file /home/jiling/.julia/compiled/v1.9/SentinelArrays/uMYVe_zIiTQ.so. From worker 3: └ @ Base loading.jl:1673 From worker 8: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] From worker 8: │ exception = Required dependency Preferences [21216c6a-2e73-6563-6e65-726566657250] failed to load from a cache file. From worker 8: └ @ Base loading.jl:1673 From worker 2: ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] From worker 2: │ exception = Required dependency Preferences [21216c6a-2e73-6563-6e65-726566657250] failed to load from a cache file. ``` might be related to #48057 and #48039

yeah, something is funky with Julia 1.9 and above