SharedArrays crashes on Julia 1.3 Windows

General Usage
1

Even though people will always tell you to upgrade to the latest version, this is the reason I hate updating old but working stuff… So since I updated from 1.0 to 1.3 SharedArrays doesn’t work anymore.

using SharedArrays

works, but when I write

p=SharedArray{Int64}(10)

Julia hangs but seems to do something in the background. After some time I’m done waiting and will abort with ctrl+c to get this error

ERROR: LoadError: InterruptException:

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:300
 [2] macro expansion at .\task.jl:319 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\macros.jl:217
 [4] top-level scope at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\macros.jl:201
 [5] include at .\boot.jl:328 [inlined]
 [6] include_relative(
SYSTEM (REPL): showing an error caused an error
ERROR: InterruptException:
Stacktrace:
 [1] sigatomic_end at .\c.jl:425 [inlined]
 [2] uv_write(::Base.TTY, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:931
 [3] unsafe_write(::Base.TTY, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:1007
 [4] unsafe_write(::REPL.Terminals.TTYTerminal, ::Ptr{UInt8}, ::UInt64) at .\io.jl:310 (repeats 2 times)
 [5] macro expansion at .\gcutils.jl:91 [inlined]
 [6] write at .\strings\io.jl:186 [inlined]
 [7] print at .\strings\io.jl:188 [inlined]
 [8] #with_output_color#707(::Bool, ::typeof(Base.with_output_color), ::Function, ::Symbol, ::IOContext{REPL.Terminals.TTYTerminal}, ::String) at .\util.jl:383
 [9] #with_output_color at .\none:0 [inlined]
 [10] #printstyled#708 at .\util.jl:397 [inlined]
 [11] #printstyled at .\none:0 [inlined]
 [12] show_tuple_as_call(::IOContext{REPL.Terminals.TTYTerminal}, ::Symbol, ::Type) at .\show.jl:1522
 [13] show_spec_linfo(::IOContext{REPL.Terminals.TTYTerminal}, ::Base.StackTraces.StackFrame) at .\stacktraces.jl:262
 [14] #show#9(::Bool, ::typeof(show), ::IOContext{REPL.Terminals.TTYTerminal}, ::Base.StackTraces.StackFrame) at .\stacktraces.jl:272
 [15] #show at .\none:0 [inlined]
 [16] #show_trace_entry#673(::String, ::typeof(Base.show_trace_entry), ::IOContext{REPL.Terminals.TTYTerminal}, ::Base.StackTraces.StackFrame, ::Int64) at .\errorshow.jl:483
 [17] (::Base.var"#kw##show_trace_entry")(::NamedTuple{(:prefix,),Tuple{String}}, ::typeof(Base.show_trace_entry), ::IOContext{REPL.Terminals.TTYTerminal}, ::Base.StackTraces.StackFrame, ::Int64) at .\none:0
 [18] show_backtrace(::IOContext{REPL.Terminals.TTYTerminal}, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at .\errorshow.jl:586
 [19] #showerror#656(::Bool, ::typeof(showerror), ::IOContext{REPL.Terminals.TTYTerminal}, ::CompositeException, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at .\errorshow.jl:79
 [20] (::Base.var"#kw##showerror")(::NamedTuple{(:backtrace,),Tuple{Bool}}, ::typeof(showerror), ::IOContext{REPL.Terminals.TTYTerminal}, ::CompositeException, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at .\none:0
 [21] #showerror#659(::Bool, ::typeof(showerror), ::IOContext{REPL.Terminals.TTYTerminal}, ::LoadError, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at .\errorshow.jl:85
 [22] (::Base.var"#kw##showerror")(::NamedTuple{(:backtrace,),Tuple{Bool}}, ::typeof(showerror), ::IOContext{REPL.Terminals.TTYTerminal}, ::LoadError, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at .\none:0
 [23] show_exception_stack(::IOContext{REPL.Terminals.TTYTerminal}, ::Array{Any,1}) at .\errorshow.jl:649
 [24] display_error(::IOContext{REPL.Terminals.TTYTerminal}, ::Array{Any,1}) at .\client.jl:102
 [25] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [26] invokelatest at .\essentials.jl:708 [inlined]
 [27] print_response(::IO, ::Any, ::Bool, ::Bool, ::Any) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:151
 [28] print_response(::REPL.AbstractREPL, ::Any, ::Bool, ::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:141
 [29] (::REPL.var"#do_respond#38"{Bool,REPL.var"#48#57"{REPL.LineEditREPL,REPL.REPLHistoryProvider},REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:719
 [30] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [31] invokelatest at .\essentials.jl:708 [inlined]
 [32] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\LineEdit.jl:2306
 [33] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:1045
 [34] run_repl(::REPL.AbstractREPL, ::Any) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:201
 [35] (::Base.var"#770#772"{Bool,Bool,Bool,Bool})(::Module) at .\client.jl:382
 [36] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [37] invokelatest at .\essentials.jl:708 [inlined]
 [38] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at .\client.jl:366
 [39] exec_options(::Base.JLOptions) at .\client.jl:304
 [40] _start() at .\client.jl:460

and some other times (not always the same error)

ERROR: InterruptException:
Stacktrace:
 [1] poptaskref(::Base.InvasiveLinkedListSynchronized{Task}) at .\task.jl:660
 [2] wait() at .\task.jl:667
 [3] wait(::Base.GenericCondition{ReentrantLock}) at .\condition.jl:106
 [4] take_buffered(::Channel{Any}) at .\channels.jl:400
 [5] take!(::Channel{Any}) at .\channels.jl:394
 [6] take!(::Distributed.RemoteValue) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:595
 [7] #remotecall_fetch#145(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:386
 [8] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:382
 [9] #remotecall_fetch#148(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Int64, ::Distributed.RRID, ::Vararg{Any,N} where N) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:417
 [10] remotecall_fetch at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:417 [inlined]
 [11] call_on_owner at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:490 [inlined]
 [12] wait at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\remotecall.jl:511 [inlined]
 [13] #_#3(::Bool, ::Array{Int64,1}, ::Type{SharedArray{Int64,1}}, ::Tuple{Int64}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:137
 [14] SharedArray at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:105 [inlined]
 [15] #_#10 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:161 [inlined]
 [16] SharedArray at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:161 [inlined]
 [17] #_#13 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:167 [inlined]
 [18] SharedArray{Int64,N} where N(::Int64) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\SharedArrays\src\SharedArrays.jl:167
 [19] top-level scope at REPL[4]:1

Since I’m already here: Is SharedArrays the prefered option when parallelizing for loops?

2

Works for me on 1.3.0 (and is quick <<1s). But you should file an issue as this should just work.

Probably you should just use threads? Then you don’t need shared-arrays. Multi-Threading · The Julia Language

1 Like
3

This is said to be still experimental. Do you have an example of how to use it without too much overhead? It seems to be more complicated than just creating a “SharedArrays” that is accessible to all workers.

4

I followed this: Announcing composable multi-threaded parallelism in Julia

I think it should be pretty simple to port the shared-array code, just use Threads.@threads for ... for you for-loops. If you used map, then use https://github.com/baggepinnen/ThreadTools.jl.

Note that you don’t use workers but instead start julia with JULIA_NUM_THREADS=4 ./julia.

2 Likes
5

" Note that you don’t use workers but instead start julia with JULIA_NUM_THREADS=4 ./julia ."

What is the difference with workers and threads?

Also what do you mean by “start” julia with… I start Julia in windows by clicking on the button and not via console.

BTW: Can you try

using Distributed
addprocs(4)
p=SharedArray{Int64}(100)

because without addprocs() Sharedarray works too

6

This works (on Linux):

julia> using Distributed, SharedArrays

julia> addprocs(4)
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> p=SharedArray{Int64}(100)
100-element SharedArray{Int64,1}:
 0

Workers are separate julia processes. Thus, e.g., you have to make sure all packages are loaded on all workers with @everywhere SomePkg. Threads run all on the same julia process but in separate threads, which can run in parallel.

Sorry, I’ve never started Julia by clicking, so I couldn’t say.

7

But does this actually mean that different threads use different processors? Say I have 4 cores. I add 4 procs because each proc runs on a different core/processor. I’m not sure this is also what happends with threads. I mean I can open actually open 20 workers with just 4 cores; doesn’t mean I get any benefits in speed.

Also: Are you sure about @everywhere SharedArrays

using SharedArrays loads on all processes or?

8

Yep, it does (if you started julia with a number of threads). Although the operating system will do the scheduling, so it will depend on what other things run, etc…

Ok, yes using might automatically do this, so, a better example is @everywhere myfun(x,y) = ...

9

This is really annoying!

Internal error: encountered unexpected error in runtime:
InterruptException()
jl_mutex_unlock at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\locks.h:138 [inlined]
jl_typeinf_end at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2587
ERROR:       From worker 5:     WARNING: Force throwing a SIGINTInterruptException:

Stacktrace:
      From worker 4:    WARNING: Force throwing a SIGINT [1]
#call#3(      From worker 3:    WARNING: Force throwing a SIGINT::
Bool,       From worker 2:      WARNING: Force throwing a SIGINT::
Array{Int64,1}, ::Type{SharedArray{Int64,1}}, ::Tuple{Int64}) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:155
 [2] Type at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:105 [inlined]
 [3] #call#10 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:161 [inlined]
 [4] Worker 4 terminated.Type
 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:161Worker 3 terminated. [inlined]

 [5] Worker 2 terminated.#call#13
 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:167Worker 5 terminated. [inlined]

 [6] SharedArray{Int64,N} where N(::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\SharedArrays\src\SharedArrays.jl:167
 [7] top-level scope at none:0
10

that sounds really frustrating. But it’s more helpful if you create a reproducible example for the problem. It will enable others to help you.

11

Actually I’m done…I’m removing Julia alltogether. I just don’t know what to search for (except Julia) in regedit to delete any important reminiscent reference that might be remembered upon any future reinstallation.

What am I supposed to do to create a reproducible example?

It happens only upon calling

using Distributed, SharedArrays
addprocs(4)
p=SharedArray{Int64}(1)

It works without calling addprocs(4).

12

Could you list out detail about your computer (cpu, Windows version, etc.), how Julia was installed on your computer, and any anti-virus that is running on your computer? Note: I can’t help troubleshoot issues on Windows, but these kind of details might help others who can.

13

from Julia 1.3 on Windows, that code works normally. can you reproduce this on a clean installation? maybe trying to execute Julia with admin privileges?

14

I’ve been uninstalling and reinstalling julia (1.3 and 1.05) numerous times with no remedy.

I then tried to uninstall everything (deleting every single path I could find) and tried to delete every regedit entry I could find. I then restarted my computer and installed 1.3 again. All of a sudden I was asked by Avast again to put julia.exe to the exception list and now it works again. So it might have been possible that it (oddly enough only the workers) was blocked by the AntiVirus program but it didn’t say anything, maybe because an older installation in the registry was still making it seem like if the program was on the exception list when the new installation julia.exe was not. I hope this makes sense somehow…

I’m on Windows10 with 4core P5 2.7GHz.

3 Likes