Hi, I need to implement the following architecture: my master worker receives some requests (via HTTP interface) validates the requests, makes some convertations and puts requests into a queue. Request from the queue is sent to the available worker where computations take place (do_job) and results are then sent back to the master worker and then to some output (E.x. CSV). I am also thinking about requests prioritization but for now FIFO queue is fine.
RemoteChannel seems to be the closest solution. I can set up two channels
const requests = Distributed.RemoteChannel(()->Channel{Request}(Inf))
const results = Distributed.RemoteChannel(()->Channel{Result}(Inf))
const worker_pool = WorkerPool(workers())
and use the following functions to put! requests and schedule them with remote_do do_job:
function sched_request(req::Request)
put!(requests, req)
@async remote_do(do_job, worker_pool)
println("task scheduled")
end
@everywhere function do_job()
req = take!(requests)
#=
do some computations -> res
=#
put!(res, results)
@async remote_do(save_as_csv, master_worker_id)
println(sent to master worker)
end
Is this RemoteChannel/put!/remote_do the proper way to deal with such workflow? In this case I have to define do_job function on all workers including master worker. However do_job is needed only on remote execution workers and save_as_csv only on master worker. Maybe there are some recommended examples of such workflow architecture?