Is it possible to load a jl file on a worker as they start?
The -L
flag does not seems to work.
using Distributed
open("worker_startup.jl", "w") do io
print(io, """
using Distributed
id = Distributed.myid()
println("worker \$(id) started !!")
""")
close(io)
end
addprocs(1; exeflags="-L worker_startup.jl")
@everywhere @show id
The point of this is that if I have an @async
task that starts workers on a cluster, I can start a pmap
call without waiting for all the workers to start.
Hoping that newly added workers will load their startup file, and then join the worker pool of the already running pmap
call
task = @async for k=1:10
sleep(1)
addprocs(1; exeflags="-L worker_startup.jl")
end
pmap(q->(sleep(1); rand()+myid()), 1:10)