Spawning inside a module

General Usage
1

Hello everyone.

I am trying to create a struct which stores a function with the intention to be executed in parallel over multiple cores. The problem seems I am facing is that struct fields seem to only reference functions defined at the process where the struct itself is defined. That in ordinary cases probably is resolved by initializing the struct on multiple workers and thus then do the computation. However, that does not work when tasks are not known in advance but in a sense are learned in a feedback loop.

So I thought I could work around the issue by storing a spawning function at the field instead. That does work excellent until I do wrap code in the module, like this:

module Test

using Distributed

struct Foo
    dispatch::Function

    function Foo(f)
        new(x->@spawn f(x))
    end
end

function calculate(foo::Foo)
    work = []

    ### Creating work
    for x in 1:10
        w = foo.dispatch(x)
        push!(work,w)
    end

    ### Fetching results
    [fetch(w) for w in work]
end

end

using Distributed

@everywhere f(x) = x^2

foo = Test.Foo(f)
Test.calculate(foo)

For that, I get a rather cryptic error message.

ERROR: LoadError: On worker 2:
UndefVarError: Test not defined
deserialize_module at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:834
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:764
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:703
deserialize_datatype at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:1050
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:743
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:703
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:750
deserialize_msg at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:703
#invokelatest#1 at ./essentials.jl:697 [inlined]
invokelatest at ./essentials.jl:696 [inlined]
message_handler_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:160
process_tcp_streams at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:117
#105 at ./task.jl:259
Stacktrace:
 [1] #remotecall_fetch#149(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Distributed.RRID) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:379
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:371
 [3] #remotecall_fetch#152 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:406 [inlined]
 [4] remotecall_fetch at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:406 [inlined]
 [5] call_on_owner at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:479 [inlined]
 [6] fetch(::Future) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:511
 [7] iterate at ./generator.jl:47 [inlined]
 [8] collect(::Base.Generator{Array{Any,1},typeof(fetch)}) at ./array.jl:619
 [9] calculate(::Main.Test.Foo) at /home/janiserdmanis/BtSync/Projects/Julia/Spawn/spawnmodule.jl:23
 [10] top-level scope at none:0
 [11] include at ./boot.jl:317 [inlined]
 [12] include_relative(::Module, ::String) at ./loading.jl:1041
 [13] include(::Module, ::String) at ./sysimg.jl:29
 [14] include(::String) at ./client.jl:388
 [15] top-level scope at none:0

Seems that worker is trying to look for function f in a Test module, while it is defined in the Main. What is also interesting is that if I do the spawning from the Main such code works excellent:

  
module Test

using Distributed

struct Foo
    dispatch::Function
end

function calculate(foo::Foo)
    work = []

    ### Creating work
    for x in 1:10
        w = foo.dispatch(x)
        push!(work,w)
    end

    ### Fetching results
    [fetch(w) for w in work]
end

end

using Distributed

@everywhere f(x) = x^2

foo = Test.Foo(x->@spawn f(x))
Test.calculate(foo)

Thus I am confused, why spawning does not work inside the module.

1 Like
2

Did you ever figure this out? I am running into the same issue and cannot find any information on what causes this behavior.

3

I found that a decent workaround is to start tasks in the constructor of the type that waits for new data from a channel and then puts those tasks as the fields of the type. To end them, I passed nothing as a special value. I used this method for my package TaskMaster - https://github.com/akels/TaskMaster.jl/blob/master/src/workmaster.jl Works pretty great for my case :wink: