I am playing around with the parallel functionality in Julia and trying to make a specific worker load a big feather file. Surprisingly this works, since fetch returns a DataFrame:
using Distributed
addprocs(4)
@everywhere using Feather
@everywhere function ReadData(path)
data = Feather.read(path)
return data
end
c = @spawnat 1 ReadData("datapath")
result=fetch(c)
If I change the @spawnat macro to worker 2, 3 or 5 like this:
c = @spawnat 2 ReadData("datapath")
I get the following error:
Error: Fatal error on process 4
│ exception =
│ IOError: write: invalid argument (EINVAL)
│ Stacktrace:
│ [1] uv_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:953
│ [2] unsafe_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:1007
│ [3] unsafe_write at ./io.jl:593 [inlined]
│ [4] macro expansion at ./gcutils.jl:91 [inlined]
│ [5] write at ./io.jl:616 [inlined]
│ [6] serialize_array_data at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:246 [inlined]
│ [7] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{UInt8,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:263
│ [8] serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:629
│ [9] serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:608
│ [10] serialize_msg(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Distributed.ResultMsg) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:90
│ [11] #invokelatest#1 at ./essentials.jl:709 [inlined]
│ [12] invokelatest at ./essentials.jl:708 [inlined]
│ [13] send_msg_(::Distributed.Worker, ::Distributed.MsgHeader, ::Distributed.ResultMsg, ::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:185
│ [14] send_msg_now at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:130 [inlined]
│ [15] send_msg_now(::Sockets.TCPSocket, ::Distributed.MsgHeader, ::Distributed.ResultMsg) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:125
│ [16] deliver_result(::Sockets.TCPSocket, ::Symbol, ::Distributed.RRID, ::Arrow.Primitive{Float64}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:111
│ [17] macro expansion at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:302 [inlined]
│ [18] (::Distributed.var"#107#109"{Distributed.CallMsg{:call_fetch},Distributed.MsgHeader,Sockets.TCPSocket})() at ./task.jl:333
└ @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:115
Worker 4 terminated.ERROR:
LoadError: ProcessExitedException(4)
Stacktrace:
[1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./task.jl:611
[2] wait() at ./task.jl:668
[3] wait(::Base.GenericCondition{ReentrantLock}) at ./condition.jl:106
[4] take_buffered(::Channel{Any}) at ./channels.jl:400
[5] take!(::Channel{Any}) at ./channels.jl:394
[6] take!(::Distributed.RemoteValue) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:595
[7] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:386
[8] #remotecall_fetch#148 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
[9] remotecall_fetch at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
[10] call_on_owner at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:490 [inlined]
[11] fetch(::Future) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:529
[12] top-level scope at /Users/merit/Documents/JuliaProject/Test.jl:19
in expression starting at /Users/merit/Documents/JuliaProject/Test.jl:19
Anybody has an idea why that is?
Thanks in advance!
Best,
Merit