I am trying to pin my julia workers to specific CPU cores with ClusterManagers.jl. However, it seems LocalAffinityManager does not work well with SharedArrays.
using Distributed, SharedArrays, ClusterManagers
addprocs(LocalAffinityManager(;affinities=collect(0:63)))
results = [SharedArray{Float64}(1024) for _ in 1:nworkers()]
@everywhere function do_something!(result)
result[:] += rand(1024)
end
for i in 1:nworkers()
remotecall_fetch(do_something!, i, results[i])
end
The above script results in
ERROR: LoadError: On worker 2:
BoundsError: attempt to access 0-element Array{Float64,1} at index [1]
getindex at ./array.jl:809 [inlined]
getindex at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/SharedArrays/src/SharedArrays.jl:508 [inlined]
macro expansion at ./multidimensional.jl:772 [inlined]
macro expansion at ./cartesian.jl:64 [inlined]
macro expansion at ./multidimensional.jl:767 [inlined]
_unsafe_getindex! at ./multidimensional.jl:762 [inlined]
_unsafe_getindex at ./multidimensional.jl:757
_getindex at ./multidimensional.jl:743 [inlined]
getindex at ./abstractarray.jl:1060 [inlined]
do_something! at /ocean/projects/phy200062p/mistguy/temp/parallel/test.jl:9
#106 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:294
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:79
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:294 [inlined]
#105 at ./task.jl:356
Stacktrace:
[1] #remotecall_fetch#143 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:394 [inlined]
[2] remotecall_fetch(::Function, ::Distributed.Worker, ::SharedArray{Float64,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:386
[3] remotecall_fetch(::Function, ::Int64, ::SharedArray{Float64,1}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
[4] remotecall_fetch(::Function, ::Int64, ::SharedArray{Float64,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/remotecall.jl:421
[5] top-level scope at /ocean/projects/phy200062p/mistguy/temp/parallel/test.jl:13
[6] include(::String) at ./client.jl:457
[7] top-level scope at REPL[1]:1
in expression starting at /ocean/projects/phy200062p/mistguy/temp/parallel/test.jl:12
If I do not use LocalAffinityManager and simply write addprocs(64), everything is fine.
So is this expected? If so, what prevents SharedArrays to work with taskset, which LocalAffinityManager uses?