# Problem using PyCall with Parallel Computing on Julia 0.6.2

**URL:** <https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745>\
**Category:** General Usage\
**Tags:** question\
**Created:** [March 15, 2018, 8:55pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745 "2018-03-15T20:55:34Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 15, 2018, 8:55pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/1 "2018-03-15T20:55:35Z")

</div>

Hi,  
I had developed a program using Julia 0.6.0 and it was working just fine, but then I updated my Julia to 0.6.2 my program started to crash. After trouble shooting and seeing some solutions on the internet my program is like this:

```julia
using PyCall 
@everywhere nx = pyimport("networkx")
@everywhere m = pyimport("matplotlib")
@everywhere m["use"]("Agg")
@everywhere plt = pyimport("matplotlib.pyplot")

```

and the error message that I get is:

```julia
WARNING: Node state is inconsistent: node 2 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 9 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 8 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 7 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 3 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 4 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 5 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 6 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
ERROR: LoadError: On worker 2:
UndefVarError: pyimport not defined

```

I stoped using `@everywhere @pyimport networkx as nx` because I would get an error message saying that `nx` was not defined.

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [March 16, 2018, 4:24am UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/2 "2018-03-16T04:24:03Z")

</div>

`@everywhere using PyCall`?

Actually; I can’t reproduce this on 0.6.2, OSX.

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 12:53pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/3 "2018-03-16T12:53:39Z")

</div>

At first I was using `@everywhere using PyCall` on 0.6.0 but on the 0.6.2 it gave me an error message saying that it could not pre compile on worker 2, and I saw in the 0.6.2 documentation that using just `using PyCall` should load the package on all workers.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 16, 2018, 2:32pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/4 "2018-03-16T14:32:18Z")

</div>

What happens if you do `using PyCall` from a single Julia process first, to ensure that it is precompiled?

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 2:39pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/5 "2018-03-16T14:39:50Z")

</div>

How is what you are suggesting from the last program that I tried? I didn’t quite understand your suggestion.

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [March 16, 2018, 3:28pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/6 "2018-03-16T15:28:10Z")

</div>

Here’s my understanding:

When you use `@everywhere using xyz` and if the package or it’s dependent packages require any compilation then all worker processses will start compiling at the same time causing a conflict for writing to the same directory.

So the solution is to trigger the compilation only once from a single process without `@everywhere`. Then, add your workers and do the normal `@everywhere using xyz` to load the library.

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 4:39pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/7 "2018-03-16T16:39:40Z")

</div>

I tried that and got the error message

```julia
ERROR: LoadError: On worker 2:
can only precompile from node 1

```

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 4:58pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/8 "2018-03-16T16:58:58Z")

</div>

And if I try to just do the imports everywhere after adding the workers I get an `ERROR: LoadError: On worker 2: UndefVarError: @pyimport not defined` and if a do the import without everywhere I get that the functions that are on the imported modules are not defined.  
`

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 16, 2018, 4:59pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/9 "2018-03-16T16:59:26Z")

</div>

> [@stevengj](#):
>
> What happens if you do using PyCall from a single Julia process first, to ensure that it is precompiled?

That is, what happens with a non-parallel Julia run, where `nworkers() == 1`?

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 5:12pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/10 "2018-03-16T17:12:18Z")

</div>

Running not parallel it works just fine

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 16, 2018, 5:35pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/11 "2018-03-16T17:35:45Z")

</div>

Could be a problem with your cluster filesystem?

What happens if you do

```julia
Base.compilecache("PyCall")
using PyCall

```

to force your parallel run to compile the cache from node 1?

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 5:53pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/12 "2018-03-16T17:53:48Z")

</div>

I got

```julia
ERROR: LoadError: On worker 2:
UndefVarError: @pyimport not defined

```

using

```julia
addWorkersCluster()
Base.compilecache("PyCall")
using PyCall

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 16, 2018, 6:06pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/13 "2018-03-16T18:06:46Z")

</div>

> `UndefVarError: @pyimport not defined`

I’m confused, since the code you posted doesn’t even call `@pyimport` (the PyCall module itself never uses it). Is there some code you are running that you aren’t posting?

---

<div class="post-metadata">

**Author:** ![dompas](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dompas](https://discourse.julialang.org/u/dompas)\
**Post date:** [March 16, 2018, 6:59pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/14 "2018-03-16T18:59:23Z")

</div>

> [@dompas](#):
>
> @everywhere nx = pyimport(“networkx”)

In this part I can also use `@pyimport networkx as nx` I am constantly trying both to see if any of them work. But when I use the one that I posted first I get the error message

```julia
ERROR: LoadError: On worker 2:
UndefVarError: pyimport not defined

```

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [April 15, 2018, 2:56pm UTC](https://discourse.julialang.org/t/problem-using-pycall-with-parallel-computing-on-julia-0-6-2/9745/15 "2018-04-15T14:56:02Z")

</div>

Could be related to [https://github.com/JuliaLang/julia/issues/26805](https://github.com/JuliaLang/julia/issues/26805) and could possibly be fixed by [https://github.com/JuliaLang/julia/pull/26813](https://github.com/JuliaLang/julia/pull/26813)?