Making code available to workers

General Usage
, ,
1

I tried to make some code, that queries information worker processes, reusable inside a module. Why does the code without a module work just fine, while nested in and exported from a module it does not?

# nomodule.jl
using Distributed

@everywhere using LinearAlgebra

function fetch_worker_info()
    info = () -> (id=myid(), hn=gethostname(), ap=Base.active_project(), bt=BLAS.get_num_threads())
    remotecall_fetch.(info, workers())
end

foreach(println, fetch_worker_info())

$ julia -p3 nomodule.jl
(id = 2, hn = "Jonas-MBP.fritz.box", ap = "/Users/jonas/Desktop/julia-code-availability/Project.toml", bt = 1)
(id = 3, hn = "Jonas-MBP.fritz.box", ap = "/Users/jonas/Desktop/julia-code-availability/Project.toml", bt = 1)
(id = 4, hn = "Jonas-MBP.fritz.box", ap = "/Users/jonas/Desktop/julia-code-availability/Project.toml", bt = 1)

However, when wrapping the function inside a module, the workers complain about not knowing that module. I expected that they didn’t need to because info, the only function they had to execute, was a lambda that is being sent to them?

# module.jl
using Distributed

@everywhere using LinearAlgebra

module Foo
using Distributed
using LinearAlgebra

function fetch_worker_info()
    info = () -> (id=myid(), hn=gethostname(), ap=Base.active_project(), bt=BLAS.get_num_threads())
    remotecall_fetch.(info, workers())
end

export fetch_worker_info
end

using .Foo

foreach(println, fetch_worker_info())

$ julia -p3 module.jl
ERROR: LoadError: On worker 2:
UndefVarError: Foo not defined
Stacktrace:
  [1] deserialize_module
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:965
  [2] handle_deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:864
  [3] deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
  [4] deserialize_datatype
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1287
  [5] handle_deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:835
  [6] deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
  [7] handle_deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:842
  [8] deserialize
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782 [inlined]
  [9] deserialize_msg
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/messages.jl:87
 [10] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [11] invokelatest
    @ ./essentials.jl:706 [inlined]
 [12] message_handler_loop
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:169
 [13] process_tcp_streams
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:126
 [14] #99
    @ ./task.jl:411
Stacktrace:
  [1] #remotecall_fetch#143
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:394 [inlined]
  [2] remotecall_fetch(::Function, ::Distributed.Worker)
    @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:386
  [3] remotecall_fetch(::Function, ::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
  [4] remotecall_fetch
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421 [inlined]
  [5] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
  [6] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
  [7] getindex
    @ ./broadcast.jl:575 [inlined]
  [8] copy
    @ ./broadcast.jl:922 [inlined]
  [9] materialize
    @ ./broadcast.jl:883 [inlined]
 [10] fetch_worker_info()
    @ Main.Foo ~/Desktop/julia-code-availability/module.jl:11
 [11] top-level scope
    @ ~/Desktop/julia-code-availability/module.jl:19
in expression starting at /Users/jonas/Desktop/julia-code-availability/module.jl:19

Is this a bug? Do I get something wrong yet again? Both?

Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_PROJECT = @.
  JULIA_PKG_PRECOMPILE_AUTO = 0
2

Try this:

@everywhere module Foo

end
3

That works, but I don’t understand why the workers should know about that module. Note that in nomodule.jl the workers also don’t know about the fetch_worker_info function.