Multithreading unstable with JuMP

JohnZ · November 3, 2020, 11:48pm

JohnZ:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x391e0d0 -- (julia) realloc: Invalid argument

signal (22): SIGABRT
port with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

Errors similar to the one I get has been reported by others. I think it may be a bug in multithreading.

github.com/JuliaLang/julia

Fatal error on multi-threaded code processing image files

opened 02:48PM - 17 Aug 19 UTC

nlw0

multithreading

The following script processes a couple of images with some file I/O. Running it… with `@threads` causes a failure. It always breaks when calling the script directly with `JULIA_NUM_THREADS>1` , though when `include`ing it in a running session it seems it breaks the first time and later it works. Am I wrong in assuming this code is thread-safe in the first place, or could this be a bug somewhere? ```julia using FileIO using Images imgs = [ "https://images.unsplash.com/photo-1554266183-2696fdafe3ff?ixlib=rb-1.2.1&auto=format&fit=crop&w=564&q=80" "https://images.unsplash.com/photo-1527026950045-9e066846bae4?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1419&q=80" "https://images.unsplash.com/photo-1549287748-f095932c9f81?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=606&q=80" "https://images.unsplash.com/photo-1437448317784-3a480be9571e?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=634&q=80" ] files = download.(imgs) Threads.@threads for f in files a = load(f) r = rand(eltype(a), size(a)...) a .= map(clamp01nan, a + 0.5*r) save(f*".png", a) end ``` The error: ``` $ JULIA_NUM_THREADS=4 julia mycode.jl Error encountered while loading "/tmp/jl_5OG85C". Fatal error: Error encountered while loading "/tmp/jl_hZpwGF". Fatal error: Error encountered while loading "/tmp/jl_BlmZ9H". Fatal error: ERROR: LoadError: concurrency violation detected Stacktrace: [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./task.jl:552 [2] wait() at ./task.jl:609 [3] wait(::Base.GenericCondition{Base.Threads.SpinLock}) at ./condition.jl:107 [4] wait(::Task) at ./task.jl:214 [5] top-level scope at ./threadingconstructs.jl:75 [6] include at ./boot.jl:328 [inlined] [7] include_relative(::Module, ::String) at ./loading.jl:1094 [8] include(::Module, ::String) at ./Base.jl:31 [9] exec_options(::Base.JLOptions) at ./client.jl:295 [10] _start() at ./client.jl:468 in expression starting at /home/user/src/julia-test/mycode.jl:13 ``` ``` julia> versioninfo() Julia Version 1.3.0-alpha.1 Commit f2513a8ca6* (2019-07-23 05:06 UTC) Platform Info: OS: Linux (x86_64-linux-gnu) CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-6.0.1 (ORCJIT, skylake) Environment: JULIA_NUM_THREADS = 4 JULIA_EDITOR = emacsclient ```

github.com/JuliaLang/julia

Julia crashes

opened 11:44AM - 21 Aug 20 UTC

closed 01:21PM - 21 Aug 20 UTC

a-ill

Julia crashes a couple times a week with the following error after I execute the… following code. It seems to happen randomly and not depend on the content of the called qml script. Code: ``` using QML using Observables load("GUI//Main.qml") exec() ``` Error: ``` Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ff8c3baade3 -- RegisterProcTableCallback at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) in expression starting at C:\Users\a_ill\Documents\GitHub\project\source\GUI.jl:4 RegisterProcTableCallback at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) RegisterProcTableCallback at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) DumpRegistryKeyDefinitions2 at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) DrvCreateLayerContext at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) wglSwapMultipleBuffers at C:\WINDOWS\SYSTEM32\OPENGL32.dll (unknown line) wglCreateLayerContext at C:\WINDOWS\SYSTEM32\OPENGL32.dll (unknown line) DumpRegistryKeyDefinitions2 at C:\WINDOWS\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_317c1bae0b648571\ig9icd64.dll (unknown line) unknown function (ip: 000000004E17BEFB) unknown function (ip: 000000004E17D9B7) unknown function (ip: 000000004E12F554) ZN14QOpenGLContext6createEv at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) ZN14QSGOpenGLLayer13updateTextureEv at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Quick.dll (unknown line) ZN14QSGOpenGLLayer13updateTextureEv at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Quick.dll (unknown line) ZN7QWindow5eventEP6QEvent at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) ZN12QQuickWindow5eventEP6QEvent at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Quick.dll (unknown line) ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Widgets.dll (unknown line) ZN12QApplication6notifyEP7QObjectP6QEvent at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Widgets.dll (unknown line) ZN16QCoreApplication20sendSpontaneousEventEP7QObjectP6QEvent at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Core.dll (unknown line) ZN22QGuiApplicationPrivate18processExposeEventEPN29QWindowSystemInterfacePrivate11ExposeEventE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) ZN22QGuiApplicationPrivate24processWindowSystemEventEPN29QWindowSystemInterfacePrivate17WindowSystemEventE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) ZN22QWindowSystemInterface22sendWindowSystemEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) ZN22QWindowSystemInterface23flushWindowSystemEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Gui.dll (unknown line) unknown function (ip: 000000004E1289AB) qt_plugin_instance at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\plugins\platforms\qwindows.dll (unknown line) unknown function (ip: 000000004E13442A) CallWindowProcW at C:\WINDOWS\System32\USER32.dll (unknown line) DispatchMessageW at C:\WINDOWS\System32\USER32.dll (unknown line) SendMessageTimeoutW at C:\WINDOWS\System32\USER32.dll (unknown line) KiUserCallbackDispatcher at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line) NtUserDispatchMessage at C:\WINDOWS\System32\win32u.dll (unknown line) DispatchMessageW at C:\WINDOWS\System32\USER32.dll (unknown line) ZN21QEventDispatcherWin3213processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Core.dll (unknown line) qt_plugin_instance at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\plugins\platforms\qwindows.dll (unknown line) ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Core.dll (unknown line) ZN16QCoreApplication4execEv at C:\Users\a_ill\.julia\artifacts\fb208e0d5ea6127e0a53e7212e75d3ca79b88b7f\bin\Qt5Core.dll (unknown line) _ZN7qmlwrap18ApplicationManager4execEv at C:\Users\a_ill\.julia\artifacts\998b10d1488e1115e652a32dab1485863cf571a7\bin\libjlqml.dll (unknown line) _ZN5jlcxx6detail11CallFunctorIvJEE5applyEPKv at C:\Users\a_ill\.julia\artifacts\998b10d1488e1115e652a32dab1485863cf571a7\bin\libjlqml.dll (unknown line) exec at C:\Users\a_ill\.julia\packages\CxxWrap\ZOkSN\src\CxxWrap.jl:590 unknown function (ip: 0000000043E80F03) jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined] do_call at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:117 eval_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:206 eval_stmt_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:157 [inlined] eval_body at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:548 jl_interpret_toplevel_thunk at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:660 jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:840 jl_parse_eval_all at /cygdrive/d/buildbot/worker/package_win64/build/src\ast.c:913 jl_load_file_string at /cygdrive/d/buildbot/worker/package_win64/build/src\ast.c:953 include_string at .\loading.jl:1088 include_string at .\loading.jl:1096 [inlined] #216 at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:174 withpath at C:\Users\a_ill\.julia\packages\CodeTools\VsjEq\src\utils.jl:30 unknown function (ip: 0000000043EC5EC4) withpath at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:9 #215 at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:171 unknown function (ip: 000000001E0B84D3) with_logstate at .\logging.jl:408 with_logger at .\logging.jl:514 [inlined] #214 at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:170 [inlined] hideprompt at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\repl.jl:127 macro expansion at C:\Users\a_ill\.julia\packages\Media\ItEPc\src\dynamic.jl:24 [inlined] evalall at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:160 jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined] do_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:655 jl_f__apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:669 [inlined] jl_f__apply_latest at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:705 #invokelatest#1 at .\essentials.jl:710 jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined] do_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:655 invokelatest at .\essentials.jl:709 jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined] do_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\builtins.c:655 macro expansion at C:\Users\a_ill\.julia\packages\Atom\ipSjf\src\eval.jl:41 [inlined] #184 at .\task.jl:356 unknown function (ip: 000000001DFF5A13) jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined] start_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:707 Allocations: 102480780 (Pool: 102441887; Big: 38893); GC: 103 ``` versioninfo: ``` Julia Version 1.5.0 Commit 96786e22cc (2020-08-01 23:44 UTC) Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-9.0.1 (ORCJIT, skylake) Environment: JULIA_EDITOR = "C:\Users\a_ill\AppData\Local\atom\app-1.50.0\atom.exe" -a JULIA_NUM_THREADS = 6 ```

github.com/JuliaLang/julia

EXCEPTION_ACCESS_VIOLATION from Flux.gradient() using Threads.@threads and Flux.gpu()

opened 07:16PM - 21 Jul 20 UTC

closed 12:09AM - 18 Mar 22 UTC

MacKenzieHnC

multithreading

I tried my absolute best looking for equivalent issues. I saw similar errors, bu…t do not know enough to understand if they are the same problem. But the output said to post this as an issue, so here I am. The idea is very simple, train some Flux models on the gpu, but have the models generated and trained from different cpu threads. As near as I can tell, this should be thread-safe to do, because everything that would get mutated in training is thread-local. The data should not be mutated by training. https://gist.github.com/MacKenzieHnC/7596910b83f7351e92a5fd5c5dcdc94d ``` # simple_double_threaded.jl using Flux using Statistics import CuArrays CuArrays.allowscalar(false) # generate a bunch of data X = rand(100) Y = 0.5X + rand(100) Xd = reduce(hcat,X) Yd = reduce(hcat,Y) # global gpu data data = gpu([(Xd,Yd)]) # cpu threads Threads.@threads for i in 1:100 # lower values only fail sometimes??? # gpu model model = gpu(Dense(1,1)) loss(x, y) = mean((model(x).-y).^2) opt = ADAM() par = params(model); for j in 1:100 # lower values only fail sometimes??? # Training crashes gloriously Flux.train!(loss,par,data,opt) end end ``` So why does it fail so dramatically? And so inconsistently when using real-world values?? Version Info: ``` julia> using InteractiveUtils julia> versioninfo() Julia Version 1.4.2 Commit 44fa15b150* (2020-05-23 18:35 UTC) Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-8.0.1 (ORCJIT, skylake) Environment: JULIA_DEPOT_PATH = C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1;C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\Julia-1.4.2\local\share\julia;C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\Julia-1.4.2\share\julia JULIA_EDITOR = "C:\Users\Lepre\AppData\Local\JuliaPro 1.4.2-1\app-1.47.0\atom.exe" -a JULIA_NUM_THREADS = 6 JULIA_PKG_SERVER = pkg.juliacomputing.com ``` And then this is the output: ``` Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- g report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks. Exception: Please submit a bug repor at 0x668432c5 -- at 0x668432c5 -- g report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION ve/d/buildbotin expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 trampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined] jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 in expression starting at C:\Users\Leprein expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 ety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATIONtrampoline_alloc atrampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined] jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 ety). Thanks. Exception: EXCEPTION_ACCESS_VIOLATION at 0x668432c5 -- at 0x668432c5 -- OLATION ve/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined] jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 trampoline_alloc at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:197 [inlined] in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threadjl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 #launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupjl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 #launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined] lined] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 ] lined] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupan#launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlinmacro expansion at C:\Users\Lepre\.juliamacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined] #mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83 mapreducedim!##kw at C:\Users\Lepre\.julmapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81 #mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83 mapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81 unknown function (ip: 000000006535BC85) apro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highljl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 in expression starting at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:18 #mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highljl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 jl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 el.jl:83 unknown function (ip: 000000006535BC85) unknown function (ip: 000000006535BC85) re\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreducejl_get_cfunction_trampoline at /cygdrive/d/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:329 #launch_configuration#578 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:61 [inlined] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:unknown function (ip: 000000006537CC8C) e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006535BC85) #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62 #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62 5 [i#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:83 mapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81 unknown function (ip: 000000006535BC85) #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62 #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62 81 ned] #mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] mapreduce at C:\Users\Lepre\.juliapro\Jumapreduce at C:\Users\Lepre\.juliapro\Ju_mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:34 _mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:34 [inlined] #mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] mapreduce at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] _sum at .\reducedim.jl:657 [inlined] _sum at .\reducedim.jl:656 [inlined] #sum#583 at .\reducedim.jl:652 [inlined] sum at .\reducedim.jl:652 [inlined] _mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined] #mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] #adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined] adjoint at .\none:0 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined] adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] #15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 d] lined] #mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] _sum at .\reducedim.jl:679 [inlined] _sum at .\reducedim.jl:678 [inlined] #sum#583 at .\reducedim.jl:652 [inlined] sum##kw at .\reducedim.jl:652 [inlined] #accum_sum#1083 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:42 [inlined] accum_sum##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:42 [inlined] unbroadcast at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:53 unbroadcast at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:53 [gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined] macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined] #train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81 train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined] #373#threadsfor_fun at .\threadingconstructs.jl:61 #373#threadsfor_fun at .\threadingconstrmacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\src\mapreduce.jl:199 [inlined]_sum at .\reducedim.jl:657 [inlined] _sum at .\reducedim.jl:656 [inlined] #sum#583 at .\reducedim.jl:652 [inlined] sum at .\reducedim.jl:652 [inlined] _mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined] #mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] #adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined] adjoint at .\none:0 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined] adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] #15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:172 map at .\tuple.jl:158 [inlined] #1090 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\broadcast.jl:74 [inlined] #2457#back at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined] Dense at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\layers\basic.jl:122 [inlined] Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 Dense at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\layers\basic.jl:133 [inlined] Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.unknown function (ip: 000000003568D1C3) _v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined] ned] line#mapreducedim!#72 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevegradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53 gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53 l.jl:83 ed] #mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jPullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 #175 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:182 erface2.jl:0 28 [inlimapreducedim!##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAnative\C91oY\src\nvtx\highlevel.jl:81 unknown function (ip: 000000006535BC85) #_mapreduce#27 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:62 81 ed] mapreduce at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] _sum at .\reducedim.jl:657 [inlined] _sum at .\reducedim.jl:656 [inlined] #sum#583 at .\reducedim.jl:652 [inlined] sum at .\reducedim.jl:652 [inlined] _mean at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:160 [inlined] #mean#4 at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] mean##kw at C:\Users\julia\AppData\Local\Julia-1.4.2\share\julia\stdlib\v1.4\Statistics\src\Statistics.jl:157 [inlined] #adjoint#667 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\array.jl:278 [inlined] adjoint at .\none:0 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] loss at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:22 [inlined] adjoint at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\lib\lib.jl:179 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:47 [inlined] #15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined] _pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:172 gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:53 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined] macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined] #train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81 train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined] #373#threadsfor_fun at .\threadingconstructs.jl:61 #373#threadsfor_fun at .\threadingconstructs.jl:28 _v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inliunknown function (ip: 000000006537CC8C) unknown function (ip: 000000006537CC8C) cts.jl:28 _v1.4.2-1\dev\SIREN\test\simple_double_t#347#back at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined] #15 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:89 [inlined] Pullback at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface2.jl:0 #50 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:177 unknown function (ip: 000000006536C31A) v1.4.2-1\packages\Zygote\1GXzF\src\compiler\instart_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:687 Allocations: 216670028 (Pool: 216622719; Big: 47309); GC: 146 Allocations: 216670028 (Pool: 216622719; Big: 47309unknown function (ip: 000000003568D1C3) unknown function (ip: 000000003568D1C3) gradient at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Zygote\1GXzF\src\compiler\interface.jl:54 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:88 [inlined] macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined] #train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81 train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79 81 inlined] ed] d] d] launch_configuration##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 unknown function (ip: 000000006536812A) unknown function (ip: 000000006536812A) e\.juliapro\JuliaPro_v1.4.2-1\packages\CUDAdrv\Uc14X\src\occupancy.jl:55 ] d] #mapreduce#25 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] mapreduce##kw at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\GPUArrays\JqOUg\src\host\mapreduce.jl:28 [inlined] _sum at .\reducedim.jl:679 [inlined] _sum at .\reducedim.jl:678 [inlined] #sum#583 at .\reducedim.jl:652 [inlined] sum##kw at .\reducedim.jl:652 [inlined] #accum_sum#1083 at C:\Users\Lepre\.juliamacro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\CuArrays\YFdj7\sstart_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:687 Allocations: 216670028 (Pool: 216622719; Big: 47309); GC: 146 rc\task.c:687 s\YFdj7\src\mapreduce.jl:199 [inlined] macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Juno\tLMZd\src\progress.jl:119 [inlined] #train!#12 at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:81 train! at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\packages\Flux\Fj3bt\src\optimise\train.jl:79 macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\macro expansion at C:\Users\Lepre\.juliapro\JuliaPro_v1.4.2-1\dev\SIREN\test\simple_double_threaded.jl:28 [inlined] ```