Addprocs() does not work

mariok90 · March 30, 2021, 9:38am

When I use addprocs() julia crashes with that error message:

julia> using Distributed

julia> workers()
1-element Array{Int64,1}:
 1

julia> addprocs()
16-element Array{Int64,1}:
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17

julia>
      From worker 16:   Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 5:
      From worker 5:    Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 5:    Exception: UNKNOWN at 0x7fff4ba9d759 -- unknown function (ip: 00007fff4ba9d759)
      From worker 5:    in expression starting at none:0
      From worker 5:    unknown function (ip: 00007fff4ba9d759)
      From worker 5:    unknown function (ip: 000000006144d620)
      From worker 5:    unknown function (ip: 000000006fd436c2)
      From worker 5:    unknown function (ip: 000000006fd42bbf)
      From worker 5:    unknown function (ip: 000000000437d668)
      From worker 5:    unknown function (ip: 0000000002c2af7c)
      From worker 5:    unknown function (ip: 0000000002c2da64)
      From worker 5:    unknown function (ip: 0000000002c22a37)
      From worker 5:    unknown function (ip: 0000000002cc5f6a)
      From worker 5:    unknown function (ip: 0000000002cca020)
      From worker 5:    unknown function (ip: 0000000002ccbb15)
      From worker 5:    unknown function (ip: 0000000003cbe060)
      From worker 5:    unknown function (ip: 000000000292c042)
      From worker 5:    unknown function (ip: 00000000027ae389)
      From worker 5:    unknown function (ip: 00000000027ae472)
      From worker 5:    unknown function (ip: 00000000027ad61f)
      From worker 16:   Exception: UNKNOWN at 0x7fff4ba9d759 -- unknown function (ip: 00007fff4ba9d759)
      From worker 16:   in expression starting at none:0
      From worker 16:   unknown function (ip: 00007fff4ba9d759)
      From worker 16:   Unwind_RaiseException at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libgcc_s_seh-1.dll (unknown line)
      From worker 16:   unknown function (ip: 000000006fd436c2)
      From worker 16:   unknown function (ip: 000000006fd42bbf)
      From worker 16:   unknown function (ip: 000000000437d668)
      From worker 16:   unknown function (ip: 0000000002c2af7c)
      From worker 16:   unknown function (ip: 0000000002c2da64)
      From worker 16:   unknown function (ip: 0000000002c22a37)
      From worker 16:   unknown function (ip: 0000000002cc5f6a)
      From worker 16:   unknown function (ip: 0000000002cca020)
      From worker 16:   unknown function (ip: 0000000002ccbb15)
Worker 16 terminated.      From worker 16:      unknown function (ip: 0000000003cbe060)

      From worker 16:   unknown function (ip: 000000000292c042)Worker 5 terminated.

      From worker 16:   unknown function (ip: 00000000027ae389)
      From worker 16:   unknown function (ip: 00000000027ae472)
      From worker 16:   unknown function (ip: 00000000027ad61f)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_generate_fptr at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_get_unspecialized at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_apply_generic at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   unknown function (ip: 000000000992972a)
      From worker 16:   jl_f_tuple at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_f__apply_latest at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   unknown function (ip: 0000000009ee7c44)
      From worker 16:   unknown function (ip: 0000000009936251)
      From worker 16:   unknown function (ip: 000000000993658c)
      From worker 16:   unknown function (ip: 00000000099365ae)
      From worker 16:   jl_finish_task at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   Allocations: 829967 (Pool: 829728; Big: 239); GC: 1
      From worker 16:
      From worker 16:   Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 16:   Exception: EXCEPTION_ACCESS_VIOLATION at 0x2977418 -- unknown function (ip: 0000000002977418)
      From worker 16:   in expression starting at none:0
      From worker 16:   unknown function (ip: 0000000002977418)
      From worker 16:   unknown function (ip: 000000000297c5f1)
      From worker 16:   unknown function (ip: 000000000297db37)
      From worker 16:   unknown function (ip: 0000000002973d27)
      From worker 16:   unknown function (ip: 000000000297d35d)
      From worker 16:   unknown function (ip: 000000000292c042)
      From worker 16:   unknown function (ip: 00000000027ae389)
      From worker 16:   unknown function (ip: 00000000027ae472)
      From worker 16:   unknown function (ip: 00000000027ad61f)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_generate_fptr at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_get_unspecialized at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_apply_generic at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jlbacktrace at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jlbacktrace at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jlbacktrace at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_atexit_hook at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_exit at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_exception_handler at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   _julia_personality at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   unknown function (ip: 00007fff4e351f6e)
      From worker 16:   unknown function (ip: 00007fff4e301453)
      From worker 16:   unknown function (ip: 00007fff4e3011a4)
      From worker 16:   unknown function (ip: 00007fff4ba9d758)
      From worker 16:   Unwind_RaiseException at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libgcc_s_seh-1.dll (unknown line)
      From worker 16:   unknown function (ip: 000000006fd436c2)
      From worker 16:   unknown function (ip: 000000006fd42bbf)
      From worker 16:   unknown function (ip: 000000000437d668)
      From worker 16:   unknown function (ip: 0000000002c2af7c)
      From worker 16:   unknown function (ip: 0000000002c2da64)
      From worker 16:   unknown function (ip: 0000000002c22a37)
      From worker 16:   unknown function (ip: 0000000002cc5f6a)
      From worker 16:   unknown function (ip: 0000000002cca020)
      From worker 16:   unknown function (ip: 0000000002ccbb15)
      From worker 16:   unknown function (ip: 0000000003cbe060)
      From worker 16:   unknown function (ip: 000000000292c042)
      From worker 16:   unknown function (ip: 00000000027ae389)
      From worker 16:   unknown function (ip: 00000000027ae472)
      From worker 16:   unknown function (ip: 00000000027ad61f)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_dump_compiles at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_extern_c at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)      From worker 16:   jl_generate_fptr at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_get_unspecialized at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_apply_generic at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   unknown function (ip: 000000000992972a)
      From worker 16:   jl_f_tuple at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   jl_f__apply_latest at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   unknown function (ip: 0000000009ee7c44)
      From worker 16:   unknown function (ip: 0000000009936251)
      From worker 16:   unknown function (ip: 000000000993658c)
      From worker 16:   unknown function (ip: 00000000099365ae)
      From worker 16:   jl_finish_task at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\bin\libjulia.dll (unknown line)
      From worker 16:   Allocations: 829967 (Pool: 829728; Big: 239); GC: 1
      From worker 15:   Internal error: encountered unexpected error in runtime:
      From worker 15:   reset_page at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:1093 [inlined]
      From worker 15:   add_page at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:1125
      From worker 15:   jl_gc_pool_alloc at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:1178 [inlined]
      From worker 15:   jl_gc_alloc_ at /cygdrive/c/buildbot/worker/package_win64/build/src\julia_internal.h:277 [inlined]
      From worker 15:   jl_gc_alloc at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:3150
      From worker 15:   jl_gc_alloc_buf at /cygdrive/c/buildbot/worker/package_win64/build/src\julia_internal.h:304 [inlined]
      From worker 15:   array_resize_buffer at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:686
      From worker 15:   jl_array_grow_at_end at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:875 [inlined]
      From worker 15:   jl_array_grow_end at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:939
      From worker 15:   _growend! at .\array.jl:892 [inlined]
      From worker 15:   resize! at .\array.jl:1085 [inlined]
      From worker 15:   resize! at .\compiler/ssair\ir.jl:992
      From worker 15:   iterate at .\compiler/ssair\ir.jl:1101
      From worker 15:   ir_inline_item! at .\compiler/ssair\inlining.jl:396
      From worker 15:   batch_inline! at .\compiler/ssair\inlining.jl:552
      From worker 15:   ssa_inlining_pass! at .\compiler/ssair\inlining.jl:77 [inlined]
      From worker 15:   run_passes at .\compiler/ssair\driver.jl:138
      From worker 15:   optimize at .\compiler\optimize.jl:174
      From worker 15:   typeinf at .\compiler\typeinfer.jl:33
      From worker 15:   typeinf_ext at .\compiler\typeinfer.jl:570
      From worker 15:   typeinf_ext at .\compiler\typeinfer.jl:601
      From worker 15:   jfptr_typeinf_ext_20437.clone_1 at C:\Users\mario\AppData\Local\Programs\Julia 1.5.3\lib\julia\sys.dll (unknown line)
      From worker 15:   _jl_invoke at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2214 [inlined]
      From worker 15:   jl_apply_generic at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2398 [inlined]
      From worker 15:   jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
      From worker 15:   jl_type_infer at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:296
      From worker 15:   jl_generate_fptr at /cygdrive/c/buildbot/worker/package_win64/build/src\jitlayers.cpp:290
      From worker 15:   jl_compile_method_internal at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:1964
      From worker 15:   jl_compile_method_internal at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:1931 [inlined]
      From worker 15:   _jl_invoke at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2224 [inlined]
      From worker 15:   jl_apply_generic at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2398
      From worker 15:   jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
      From worker 15:   do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:655
      From worker 15:   #114 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:315
      From worker 15:   run_work_thunk at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:79
      From worker 15:   #113 at .\task.jl:356
      From worker 15:   unknown function (ip: 000000001965dbe3)
      From worker 15:   jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
      From worker 15:   start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:705
      From worker 15:   ReadOnlyMemoryError()

It does not matter if I use version 1.6 or 1.5.3. Any idea how to fix this?

kaslusimoes · March 30, 2021, 12:39pm

Hi,

I’m definitely not someone knowledgeable in this, but a couple of questions that might give relevant information to the clever guys around here

Does that also happen if you run addprocs(1)?
What is the output for Sys.CPU_THREADS?
Where are you running this? Which machine, which OS? Maybe give us the output of versioninfo()

mariok90 · March 30, 2021, 12:51pm

Thanks for the reply. versioninfo() yields:

Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i9-9900KF CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

Strangely, addprocs(1) also works without an error.

kaslusimoes · March 30, 2021, 12:54pm

Have you checked Sys.CPU_THREADS? On the documentation it says that calling addprocs without argument uses that value as a default, and considering that addprocs(1) works I’m thinking if maybe you’re adding more procs than you can (not sure if there’s such a thing as “can” here, though, just a guess of mine)

mariok90 · March 30, 2021, 1:07pm

julia> Sys.CPU_THREADS
16

addprocs() used to work some time ago at least…

kaslusimoes · March 30, 2021, 1:18pm

I see… Well, my guess was a long-shot anyway. Not sure what the problem might be, sorry I tried the same here but without error.

Hopefully someone else will pop in soon and help you, but perhaps in the meantime you can just find the upper bound of addprocs(x) that you can use without Julia crashing

mariok90 · March 30, 2021, 1:25pm

I just rebooted my computer and now it works again. Should have done that sooner… Thanks for your effort anyway! It was probably some Windows related issue.