Documentation about using anonymous and named functions with pmap

General Usage
1

I noticed that anonymous functions behave differently from regular functions w.r.t. serialization and code availability on other processes. e.g., when julia is run with multiple processes,

f(x) = x^2
pmap(f, 1:10)

throws an error, but

f = x->x^2
pmap(f, 1:10)

works fine. Is there documentation about the specific differences between anonymous and normal named functions that allow them to be used in this way?

(I sort of already understand that it is simpler to serialize anonymous functions because they only have 1 method, but I would like to understand the design decision so that I can make my code future-proof)

2 Likes
2

The first version should be

@everywhere f(x) = x^2
pmap(f, 1:10)

The @everywhere ensures the function can be used by every process. You can find the documentation for this here.

Not a parallel expert, but I assume the anonymous function just gets passed to every process by pmap which is why it works without the @everywhere.

1 Like
3

This is not true in general, you can add methods to anonymous functions:

julia> f = (x) -> x^2
#16 (generic function with 1 method)

julia> (::typeof(f))(x::Int) = 10x

julia> f
#16 (generic function with 2 methods) # Note that f now has 2 methods

julia> f(2.)
4.0

julia> f(2)
20
6 Likes
4

Thanks, @Karajan! Yes, I understand how to get it working (though sometimes @everywhere won’t suffice: https://github.com/timholy/ProgressMeter.jl/issues/125); I’m trying to understand why the decision to have anonymous functions behave differently was made, whether this is desired behavior that is going to carry on into the future, and what differences between anonymous and regular functions make this different behavior reasonable. (also, is there a way to tell if a function is anonymous? the output in the REPL for a function always just says “generic function”)

5

Ah, good point… one less hypothesis for why the behavior is different. Maybe it has to do with anonymous functions being nameless? There is no risk that an anonymous function will clash with a name already defined on the other process?

6

I actually can’t reproduce the error on 1.0.3. What error are you getting?

7

Were you perhaps executing the code with just one process?

julia> using Distributed; addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> f(x) = x^2
f (generic function with 1 method)

julia> pmap(f, 1:10)
ERROR: On worker 2:
UndefVarError: #f not defined

(this error is, of course, expected; the question is “why is the behavior of anonymous functions different, and will this always be the case?”)

1 Like
8

Ah yes, that wasn’t very smart of me.

9

I got similar issue when executing @spawnat.
Can I know any conclude here?

10

Another more weird case:

using Distributed
using ClusterManagers
addprocs(2)
@everywhere begin
    using FFTW
    FFTW.set_num_threads(1)
    ##
    N=round(Int, 2048)
    ##
    Tpx=plan_fft(rand(ComplexF64,N))
    ##
    function mapfunc(Tpx)
        test2->Tpx * test2
    end
    f=mapfunc(Tpx)
end
##

test2=[rand(ComplexF64,N), rand(ComplexF64,N)]
F=pmap(f,test2)
print("Completed")
##

Error message:

julia debug.jl
      From worker 3:
      From worker 3:    Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 3:    Exception: EXCEPTION_ACCESS_VIOLATION at 0x54516564 -- .text at C:\Users\dell\.julia\artifacts\b7dd1809d0626eac3bf6f97ba8ccfbb6cc63c509\bin\libfftw3-3.dll (unknown line)
      From worker 3:    in expression starting at none:0
      From worker 3:    .text at C:\Users\dell\.julia\artifacts\b7dd1809d0626eac3bf6f97ba8ccfbb6cc63c509\bin\libfftw3-3.dll (unknown line)
      From worker 3:    unsafe_execute! at C:\Users\dell\.julia\packages\FFTW\SDUwi\src\fft.jl:466 [inlined]
      From worker 3:    * at C:\Users\dell\.julia\packages\FFTW\SDUwi\src\fft.jl:721 [inlined]
      From worker 3:    #3 at F:\Simulation\Trap simulation\2layers trap\BEM2Layers\ElectronTrapPost\DC_1D\th2\debug.jl:15
      From worker 3:    unknown function (ip: 0000000052e144a6)
      From worker 3:    jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
      From worker 3:    do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:713
      From worker 3:    #106 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:278
      From worker 3:    run_work_thunk at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:63
      From worker 3:    macro expansion at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:278 [inlined]
      From worker 3:    #105 at .\task.jl:423
      From worker 3:    unknown function (ip: 0000000052e061a3)
      From worker 2:
      From worker 2:    Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 3:    jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
      From worker 3:    start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:877
      From worker 3:    Allocations: 7968162 (Pool: 7964985; Big: 3177); GC: 10
      From worker 2:    Exception: EXCEPTION_ACCESS_VIOLATION at 0x544a6564 -- .text at C:\Users\dell\.julia\artifacts\b7dd1809d0626eac3bf6f97ba8ccfbb6cc63c509\bin\libfftw3-3.dll (unknown line)
      From worker 2:    in expression starting at none:0
      From worker 2:    .text at C:\Users\dell\.julia\artifacts\b7dd1809d0626eac3bf6f97ba8ccfbb6cc63c509\bin\libfftw3-3.dll (unknown line)
      From worker 2:    unsafe_execute! at C:\Users\dell\.julia\packages\FFTW\SDUwi\src\fft.jl:466 [inlined]
      From worker 2:    * at C:\Users\dell\.julia\packages\FFTW\SDUwi\src\fft.jl:721 [inlined]
      From worker 2:    #3 at F:\Simulation\Trap simulation\2layers trap\BEM2Layers\ElectronTrapPost\DC_1D\th2\debug.jl:15
      From worker 2:    unknown function (ip: 0000000052da38a6)
      From worker 2:    jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
      From worker 2:    do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:713
      From worker 2:    #106 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:278
      From worker 2:    run_work_thunk at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:63
      From worker 2:    macro expansion at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:278 [inlined]
      From worker 2:    #105 at .\task.jl:423
      From worker 2:    unknown function (ip: 0000000052d958a3)
      From worker 2:    jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
      From worker 2:    start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:877
      From worker 2:    Allocations: 7966745 (Pool: 7963569; Big: 3176); GC: 10
Worker 2 terminated.
ERROR: LoadError: Worker 3 terminated.ProcessExitedException
(2)
Stacktrace:
  [1] (::Base.var"#892#894")(x::Task)
    @ Base .\asyncmap.jl:177
  [2] foreach(f::Base.var"#892#894", itr::Vector{Any})
    @ Base .\abstractarray.jl:2694
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::Vector{Vector{ComplexF64}})
    @ Base .\asyncmap.jl:177
  [4] wrap_n_exec_twice
    @ .\asyncmap.jl:153 [inlined]
  [5] #async_usemap#877
    @ .\asyncmap.jl:103 [inlined]
  [6] #asyncmap#876
    @ .\asyncmap.jl:81 [inlined]
  [7] pmap(f::Function, p::WorkerPool, c::Vector{Vector{ComplexF64}}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{Any}, retry_check::Nothing)
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\pmap.jl:126
  [8] pmap(f::Function, p::WorkerPool, c::Vector{Vector{ComplexF64}})
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\pmap.jl:101
  [9] pmap(f::Function, c::Vector{Vector{ComplexF64}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\pmap.jl:156
 [10] pmap(f::Function, c::Vector{Vector{ComplexF64}})
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\pmap.jl:156
 [11] top-level scope
    @ F:\Simulation\Trap simulation\2layers trap\BEM2Layers\ElectronTrapPost\DC_1D\th2\debug.jl:19
in expression starting at F:\Simulation\Trap simulation\2layers trap\BEM2Layers\ElectronTrapPost\DC_1D\th2\debug.jl:19
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#645")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base .\stream.jl:892
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base .\stream.jl:900
 [3] unsafe_read
   @ .\io.jl:724 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base .\io.jl:723
 [5] read!
   @ .\io.jl:725 [inlined]
 [6] deserialize_hdr_raw
   @ C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:165
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:126
 [9] (::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed .\task.jl:423
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base .\stream.jl:408
  [2] (::Base.var"#wait_locked#645")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base .\stream.jl:894
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base .\stream.jl:900
  [4] unsafe_read
    @ .\io.jl:724 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base .\io.jl:723
  [6] read!
    @ .\io.jl:725 [inlined]
  [7] deserialize_hdr_raw
    @ C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:165
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed C:\Users\dell\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Distributed\src\process_messages.jl:126
 [10] (::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed .\task.jl:423