Problem using PyCall with Parallel Computing on Julia 0.6.2

General Usage
1

Hi,
I had developed a program using Julia 0.6.0 and it was working just fine, but then I updated my Julia to 0.6.2 my program started to crash. After trouble shooting and seeing some solutions on the internet my program is like this:

using PyCall 
@everywhere nx = pyimport("networkx")
@everywhere m = pyimport("matplotlib")
@everywhere m["use"]("Agg")
@everywhere plt = pyimport("matplotlib.pyplot")

and the error message that I get is:

WARNING: Node state is inconsistent: node 2 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 9 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 8 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 7 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 3 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 4 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 5 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
WARNING: Node state is inconsistent: node 6 failed to load cache from /home/lps/.julia/lib/v0.6/PyCall.ji. Got:
WARNING: can only precompile from node 1
ERROR: LoadError: On worker 2:
UndefVarError: pyimport not defined

I stoped using @everywhere @pyimport networkx as nx because I would get an error message saying that nx was not defined.

2

@everywhere using PyCall?

Actually; I can’t reproduce this on 0.6.2, OSX.

3

At first I was using @everywhere using PyCall on 0.6.0 but on the 0.6.2 it gave me an error message saying that it could not pre compile on worker 2, and I saw in the 0.6.2 documentation that using just using PyCall should load the package on all workers.

4

What happens if you do using PyCall from a single Julia process first, to ensure that it is precompiled?

5

How is what you are suggesting from the last program that I tried? I didn’t quite understand your suggestion.

6

Here’s my understanding:

When you use @everywhere using xyz and if the package or it’s dependent packages require any compilation then all worker processses will start compiling at the same time causing a conflict for writing to the same directory.

So the solution is to trigger the compilation only once from a single process without @everywhere. Then, add your workers and do the normal @everywhere using xyz to load the library.

7

I tried that and got the error message

ERROR: LoadError: On worker 2:
can only precompile from node 1
8

And if I try to just do the imports everywhere after adding the workers I get an ERROR: LoadError: On worker 2: UndefVarError: @pyimport not defined and if a do the import without everywhere I get that the functions that are on the imported modules are not defined.
`

9

That is, what happens with a non-parallel Julia run, where nworkers() == 1?

10

Running not parallel it works just fine

11

Could be a problem with your cluster filesystem?

What happens if you do

Base.compilecache("PyCall")
using PyCall

to force your parallel run to compile the cache from node 1?

12

I got

ERROR: LoadError: On worker 2:
UndefVarError: @pyimport not defined

using

addWorkersCluster()
Base.compilecache("PyCall")
using PyCall
13

UndefVarError: @pyimport not defined

I’m confused, since the code you posted doesn’t even call @pyimport (the PyCall module itself never uses it). Is there some code you are running that you aren’t posting?

14

In this part I can also use @pyimport networkx as nx I am constantly trying both to see if any of them work. But when I use the one that I posted first I get the error message

ERROR: LoadError: On worker 2:
UndefVarError: pyimport not defined
15

Could be related to https://github.com/JuliaLang/julia/issues/26805 and could possibly be fixed by https://github.com/JuliaLang/julia/pull/26813?