Is it possible to load a jl file on a worker as they start?
The -L flag does not seems to work.
using Distributed
open("worker_startup.jl", "w") do io
print(io, """
using Distributed
id = Distributed.myid()
println("worker \$(id) started !!")
""")
close(io)
end
addprocs(1; exeflags="-L worker_startup.jl")
@everywhere @show id
The point of this is that if I have an @async task that starts workers on a cluster, I can start a pmap call without waiting for all the workers to start.
Hoping that newly added workers will load their startup file, and then join the worker pool of the already running pmap call
task = @async for k=1:10
sleep(1)
addprocs(1; exeflags="-L worker_startup.jl")
end
pmap(q->(sleep(1); rand()+myid()), 1:10)