"@everywhere using AAA" causes races on writing precompilation files?

It seems to me that the precompiler doesn’t lock files properly to prevent race conditions on generating precompilation files. I have a script which uses the Distributed module which I run on a single multicore computer running Linux, so all the worker threads share the same precompilation cache. The script executes many lines of the form

@everywhere using Module_AAA

and I have the distinct impression that the precompiler of v. 1.1 doesn’t like this (lots of precompilation errors are generated). If I precompile all included modules by executing a non-parallel script consisting of only the same “using” statements without @everywhere, the precompiler works ok, and then the parallel script subsequently runs without precompilation errors (since all modules are up-to-date in the precompilation cache).

You can load the module on the main process first, e.g.

using Module_AAA
@everywhere using Module_AAA

or you can try out the recent https://github.com/simonbyrne/PkgLock.jl package.

2 Likes

Thanks for the workaround.

And thanks for pointing me to that package, via its description I’ve come to understand that there are similar, if not identical, reported open issues with the precompilation system.

another workaround would be to just issue ]precompile before launching the workers. alas, there is a bug in it.