Precompilation failure: could not load from cache file

General Usage
1

Hi folks,

I’m seeing odd precompilation errors when running my application using Julia 1.1. I’m wondering what the right approach is to debug these. If I do a minimal example, I don’t see this issue, but I’m sort of stumped as to what is actually causing the precompilation to fail. It happens on a number of different packages, seemingly at random (e.g. replace ProtoBuf below with DataFrames or others).

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Application [top-level]
│   exception = Required dependency ProtoBuf [3349acd9-ac6a-5e09-bcdb-63829b23a429] failed to load from a cache file.
└ @ Base loading.jl:969
2

Turns out the issue was not the JULIA_DEPOT_PATH/cache location, though I found an issue there.

The primary problem was calling ‘@everywhere import ’ from the application entrypoint, without previously calling ‘import ’ when running with multiple workers. It seems that Julia does not serialize creating the compilation cache, with the result that the various workers attempt to clobber each other’s cache.

In addition I found that setting DEPOT_PATH needs to be done using @everywhere, otherwise the workers end up populating one cache, and the main process populates another.

MWE (replace ODBC with virtually any package):

@everywhere pushfirst!(Base.DEPOT_PATH, "/tmp/test.cache")
@everywhere using Distributed
@everywhere import ODBC

remotecall_fetch(println, 1, "Hello 1")
remotecall_fetch(println, 2, "Hello 2")

$ julia -p2 /tmp/test.jl
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│   exception = Required dependency IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d] failed to load from a cache file.
└ @ Base loading.jl:969
      From worker 2:	┌ Warning: The call to compilecache failed to create a usable precompiled cache file for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
...
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
│   exception = Required dependency StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] failed to load from a cache file.
└ @ Base loading.jl:969
Hello 1
      From worker 2:	Hello 2
1 Like