Run Parallel Python in Julia

How does PyCall work under the hood. Can I run parallel python processes within Julia?

1 Like

I can’t find anything that saids that you can’t. Please try and show us if it worked or how it errored, it would be very helpful to have info on that

That’s what I get…

julia> for i in 1:5

           Threads.@spawn py"1+1"

           println(i)

       end    

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffe3c47ec92 --  ION  with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

Exception: EXCEPTION_ACCESS_VIOLATIONEXCEPTION_ACCESS_VIOLA at 0x7ffe3c3df508 -- unknown function (ip: 00007FFE3C3DF508)

in expression starting at REPL[9]:0

unknown function (ip: 00007FFE3C3DF507)

unknown function (ip: 00007FFE3C3E1765)

unknown function (ip: 00007FFE3C3E0F11)

unknown function (ip: 00007FFE3C4D8846)

PyAST_FromNodeObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknowin expression starting at REPL[9]:0

in expression starting at REPL[9]:0

ssbauer\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

PyArena_Free at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

PyAST_FromNodeObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

PyAST_FromNodeObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknowPyAST_FromNodeObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

PyParser_ASTFromStringObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown Py_CompileStringExFlags at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

ne)

Py_CompileStringExFlags at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

Py_CompileStringObject at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

Py_CompileString at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

Py_CompileStringExFlags at C:\Users\<user_name>\AppData\Local\Programs\Python\Python38\python38.dll (unknown line)

macro expansion at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\exception.jl:81 [inlined]

pyeval_ at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\pyeval.jl:36

pyeval_ at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\pyeval.jl:36

\python38.dll (unknown line)

macro expansion at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\exception.jl:81 [inlined#6 at .\threadingconstructs.jl:113

unknown function (ip: 000000001DCBE383)

julia\packages\PyCall\ttONZ\src\exception.jl:81 [inlined]

pyeval_ at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\pyeval.jl:36

macro expansion at C:\Users\<user_name>\.julia\packages\PyCall\ttONZ\src\pyeval.jl:232 [inlined]

#6 at .\threadingconstructs.jl:113

unknown function (ip: 000000001DCBE383)

unknown function (ip: 000000001DCBE383)

_jl_invoke at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2141 [inlined]

jl_apply_generic at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2305

d]

jl_apply_generic at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2305

jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1631 [inlijl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1631 [inlined]

start_task at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\task.c:659

Allocations: 7897480 (Pool: 7896151; Big: 1329); GC: 8

Allocations: 7897480 (Pool: 7896151; Big: 1329); GC: 8

ild/src/cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1631 [inlined]

start_task at /cygdrive/d/buildbot/worker/package_win64/build/src/cygdrive/d/buildbot/worker/package_win64/build/src\task.c:659

Allocations: 7897480 (Pool: 7896151; Big: 1329); GC: 8

Not sure if this helps for your use case, but from a quick check I made I think I am able to run py"1+1" on multiple worker processes using pmap. Do you need it to run on multiple threads or processes?

julia> using Distributed

julia> addprocs(2);

julia> @everywhere using PyCall

julia> @everywhere function pyfoo()
       py"1+1"
       end

julia> pmap(x->pyfoo(), 1:2)
2-element Array{Int64,1}:
 2
 2
1 Like