# "Failed to precompile" error on worker processes

**URL:** <https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638>\
**Category:** General Usage\
**Created:** [June 2, 2020, 11:18pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638 "2020-06-02T23:18:09Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 2, 2020, 11:18pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/1 "2020-06-02T23:18:09Z")

</div>

I’m trying to run my code on multiple processes. It was working before but I updated Atom and it no longer works. I’m getting “failed to precompile” errors when I try to use @everywhere to load a package.

Any tips? The base errors seem to be IOError like “mkdir: file already exists”

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [June 2, 2020, 11:47pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/2 "2020-06-02T23:47:41Z")

</div>

Maybe each worker is trying to compile packages concurrently?

I don’t have a clean solution for this, but you could try `using Package` on the master process to precompile the package before attempting `@everywhere using Package`. If distributed, you’ll probably want to precompile on just one worker on each host machine before loading on all remote workers.

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 3, 2020, 3:01am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/3 "2020-06-03T03:01:26Z")

</div>

Thanks, it was Distributed. I ended up deleting all my local packages (.julia/packages) and their compiled (.julia/compiled) files, restarted Julia, precompiled in the main thread, then ran the workers.

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 3, 2020, 3:58am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/4 "2020-06-03T03:58:39Z")

</div>

> [@greg\_plowman](#):
>
> Maybe each worker is trying to compile packages concurrently?

Glad its fixed, although maybe its worth mentioning that it couldn’t have been as simple as just this. When you run `@everywhere using Package`, Julia is first loading/precompiling the package just on the main process, and only after that is done is it loading on the workers, exactly to avoid such a race condition (this is why it takes 2x as long to load a package when you have workers).

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [June 3, 2020, 4:32am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/5 "2020-06-03T04:32:11Z")

</div>

> [@marius311](#):
>
> Julia is first loading/precompiling the package just on the main process, and only after that is done is it loading on the workers

Yes, I haven’t had an issue with local workers on the same machine (presumably because all workers share the same compile location). But won’t a race still exist for remote workers on each remote host?

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 3, 2020, 4:42am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/6 "2020-06-03T04:42:44Z")

</div>

You’re right, thats a good point, if your remote machines didn’t have a shared filesystem and you had more than one worker per machine, those could end up precompiling simultaneously and cause problems.

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 3, 2020, 1:38pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/7 "2020-06-03T13:38:27Z")

</div>

FWIW, I was using Distributed on a local machine, and with @everywhere the workers failed with “Error: LoadError: IOError: unlink: resource busy or locked (EBUSY)” or “Cannot write cache file “C:\Users\nicho\.julia\compiled\v1.4\CategoricalArrays\RHXoP\_CqdxD.ji”” when it was trying to load the active project’s module on each worker. The second time I ran it, it worked.

```julia
      From worker 7: ERROR: LoadError: IOError: unlink: resource busy or locked (EBUSY)
      From worker 7: Stacktrace:
      From worker 7: [1] uv_error at .\libuv.jl:97 [inlined]
      From worker 7: [2] unlink(::String) at .\file.jl:888
      From worker 7: [3] rm(::String; force::Bool, recursive::Bool) at .\file.jl:268
      From worker 7: [4] create_expr_cache(::String, ::String, ::Array{Pair{Base.PkgId,UInt64},1}, ::Base.UUID) at .\loading.jl:1149
      From worker 7: [5] compilecache(::Base.PkgId, ::String) at .\loading.jl:1261
      From worker 7: [6] _require(::Base.PkgId) at .\loading.jl:1029
      From worker 7: [7] require(::Base.PkgId) at .\loading.jl:927
      From worker 7: [8] require(::Module, ::Symbol) at .\loading.jl:922
      From worker 7: [9] include(::Module, ::String) at .\Base.jl:377
      From worker 7: [10] top-level scope at none:2
      From worker 7: [11] eval at .\boot.jl:331 [inlined]
      From worker 7: [12] eval(::Expr) at .\client.jl:449
      From worker 7: [13] top-level scope at .\none:3
      From worker 7: in expression starting at C:\Users\nicho\.julia\packages\LocalizationMicroscopy\2eAcs\src\LocalizationMicroscopy.jl:3

      From worker 7: ERROR: LoadError: Failed to precompile LocalizationMicroscopy [798e6d30-19ab-11e9-36f8-37cfc5a8426e] to C:\Users\nicho\.julia\compiled\v1.4\LocalizationMicroscopy\bX0nX_CqdxD.ji.
      From worker 7: Stacktrace:
      From worker 7: [1] error(::String) at .\error.jl:33
      From worker 7: [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
      From worker 7: [3] _require(::Base.PkgId) at .\loading.jl:1029
      From worker 7: [4] require(::Base.PkgId) at .\loading.jl:927
      From worker 7: [5] require(::Module, ::Symbol) at .\loading.jl:922
      From worker 7: [6] include(::Module, ::String) at .\Base.jl:377
      From worker 7: [7] top-level scope at none:2
      From worker 7: [8] eval at .\boot.jl:331 [inlined]
      From worker 7: [9] eval(::Expr) at .\client.jl:449
      From worker 7: [10] top-level scope at .\none:3
      From worker 7: in expression starting at C:\Users\nicho\source\repos\SMLMAssociationAnalysis_NCB.jl\src\SMLMAssociationAnalysis_NCB.jl:4

```

Other workers had different errors:

```julia
      From worker 6: Cannot write cache file "C:\Users\nicho\.julia\compiled\v1.4\CategoricalArrays\RHXoP_CqdxD.ji".
      From worker 6: ERROR: LoadError: Failed to precompile CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597] to C:\Users\nicho\.julia\compiled\v1.4\CategoricalArrays\RHXoP_CqdxD.ji.
      From worker 6: Stacktrace:
      From worker 6: [1] error(::String) at .\error.jl:33
      From worker 6: [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
      From worker 6: [3] _require(::Base.PkgId) at .\loading.jl:1029
      From worker 6: [4] require(::Base.PkgId) at .\loading.jl:927
      From worker 6: [5] require(::Module, ::Symbol) at .\loading.jl:922
      From worker 6: [6] include(::Module, ::String) at .\Base.jl:377
      From worker 6: [7] top-level scope at none:2
      From worker 6: [8] eval at .\boot.jl:331 [inlined]
      From worker 6: [9] eval(::Expr) at .\client.jl:449
      From worker 6: [10] top-level scope at .\none:3
      From worker 6: in expression starting at C:\Users\nicho\.julia\packages\CSV\vyG0T\src\CSV.jl:17

      From worker 6: ERROR: LoadError: Failed to precompile CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b] to C:\Users\nicho\.julia\compiled\v1.4\CSV\HHBkp_CqdxD.ji.
      From worker 6: Stacktrace:
      From worker 6: [1] error(::String) at .\error.jl:33
      From worker 6: [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
      From worker 6: [3] _require(::Base.PkgId) at .\loading.jl:1029
      From worker 6: [4] require(::Base.PkgId) at .\loading.jl:927
      From worker 6: [5] require(::Module, ::Symbol) at .\loading.jl:922
      From worker 6: [6] include(::Module, ::String) at .\Base.jl:377
      From worker 6: [7] top-level scope at none:2
      From worker 6: [8] eval at .\boot.jl:331 [inlined]
      From worker 6: [9] eval(::Expr) at .\client.jl:449
      From worker 6: [10] top-level scope at .\none:3
      From worker 6: in expression starting at C:\Users\nicho\.julia\packages\LocalizationMicroscopy\2eAcs\src\LocalizationMicroscopy.jl:3

      From worker 6: ERROR: LoadError: Failed to precompile LocalizationMicroscopy [798e6d30-19ab-11e9-36f8-37cfc5a8426e] to C:\Users\nicho\.julia\compiled\v1.4\LocalizationMicroscopy\bX0nX_CqdxD.ji.
      From worker 6: Stacktrace:
      From worker 6: [1] error(::String) at .\error.jl:33
      From worker 6: [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
      From worker 6: [3] _require(::Base.PkgId) at .\loading.jl:1029
      From worker 6: [4] require(::Base.PkgId) at .\loading.jl:927
      From worker 6: [5] require(::Module, ::Symbol) at .\loading.jl:922
      From worker 6: [6] include(::Module, ::String) at .\Base.jl:377
      From worker 6: [7] top-level scope at none:2
      From worker 6: [8] eval at .\boot.jl:331 [inlined]
      From worker 6: [9] eval(::Expr) at .\client.jl:449
      From worker 6: [10] top-level scope at .\none:3
      From worker 6: in expression starting at C:\Users\nicho\source\repos\SMLMAssociationAnalysis_NCB.jl\src\SMLMAssociationAnalysis_NCB.jl:4

```

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 7, 2020, 7:51pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/8 "2020-06-07T19:51:33Z")

</div>

Ugh, this just happened again after doing pkg\>update.

From various workers:

ERROR: LoadError: IOError: unlink: resource busy or locked (EBUSY)  
ERROR: LoadError: LoadError: Failed to precompile GR [28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71] to C:\Users\nicho.julia\compiled\v1.4\GR\NDU5Y\_CqdxD.ji.

---

<div class="post-metadata">

**Author:** ![samuel\_okon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samuel_okon/32/10285_2.png) [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)\
**Post date:** [June 11, 2020, 11:47am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/9 "2020-06-11T11:47:03Z")

</div>

I also see this both on travis CI and on my local machine when using julia 1.4.2

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 13, 2020, 4:55am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/10 "2020-06-13T04:55:03Z")

</div>

And again… If anyone tries to use my code, they’re probably going to encounter this problem. And having 16 workers all trying to compile simultaneously stalls the system.

Any ideas where it might be coming from?

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 13, 2020, 7:12am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/11 "2020-06-13T07:12:10Z")

</div>

Could it be that the workers are running in a different environment than the main process, so the precompilation which happens on the main process doesn’t actually create the needed precompile files, and then the workers are all trying precompile in unison and you’re getting this error?

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 13, 2020, 1:18pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/12 "2020-06-13T13:18:03Z")

</div>

Hmm… I’m not sure how to check that. I’m setting `] activate .` to my REPL and initializing the workers with `addprocs(exeflags="--project")`

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 13, 2020, 7:51pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/13 "2020-06-13T19:51:16Z")

</div>

> [@BioTurboNick](#):
>
> Hmm… I’m not sure how to check that

Sounds like it wouldn’t be the case based on that, but as a sanity check

```julia
@everywhere (using Pkg; println(Pkg.API.Context().env.project_file)

```

should tell you.

> [@samuel\_okon](#):
>
> I also see this both on travis CI

If this is reproducible I recommend filing an Issue in the Julia repo, `@everywhere using Foo` with all workers on a local machine using the same filesystem should not be causing any precompile errors.

---

<div class="post-metadata">

**Author:** ![samuel\_okon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samuel_okon/32/10285_2.png) [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)\
**Post date:** [June 13, 2020, 8:28pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/14 "2020-06-13T20:28:32Z")

</div>

> [@marius311](#):
>
> If this is reproducible I recommend filing an Issue in the Julia repo, `@everywhere using Foo` with all workers on a local machine using the same filesystem should not be causing any precompile errors.

Thanks. It seems this was cased by precompilation. Precompiling on master processs before loading to worker processes works.

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 13, 2020, 8:34pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/15 "2020-06-13T20:34:42Z")

</div>

> [@samuel\_okon](#):
>
> Precompiling on master processs before loading to worker processes works.

Just to be clear, having to do this by hand is a bug, unless you’re in a somewhat unusual distributed setup (eg different environments or non-shared filesystem). `using Foo` or `@everywhere using Foo` should be doing this for you.

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 13, 2020, 8:44pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/16 "2020-06-13T20:44:16Z")

</div>

I created an issue:

> <https://github.com/JuliaLang/julia/issues/36184>
>
> When I update packages using pkg\>update, using Distributed with \`@everywhere usi…ng MyPackage\`, I will routinely encounter various errors related to precompilation emitted by the various workers.
> 
> Some examples here:
> https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/8
> 
> I don't know if this is a Pkg issue or a Distributed issue.
> 
> Sometimes manually \`using\` the problematic packages in the REPL allows things to work again.

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [June 14, 2020, 8:55am UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/17 "2020-06-14T08:55:44Z")

</div>

> [@BioTurboNick](#):
>
> Hmm… I’m not sure how to check that. I’m setting `] activate .` to my REPL and initializing the workers with `addprocs(exeflags="--project")`

Maybe also compare `DEPOT_PATH` on master and workers:

```julia
using Distributed
addprocs()
DEPOT_PATH
remotecall_fetch(()->DEPOT_PATH, 2)

```

and compare load paths:

```julia
Base.load_path()
remotecall_fetch(Base.load_path, 2)

```

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 15, 2020, 1:41pm UTC](https://discourse.julialang.org/t/failed-to-precompile-error-on-worker-processes/40638/19 "2020-06-15T13:41:05Z")

</div>

Both DEPOT\_PATH and Base.load\_path() are identical between the main process and workers.