# RemoteChannels and workload queue architecture

**URL:** <https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056>\
**Category:** General Usage\
**Tags:** parallel\
**Created:** [November 27, 2018, 11:26am UTC](https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056 "2018-11-27T11:26:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [November 27, 2018, 11:26am UTC](https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056/1 "2018-11-27T11:26:29Z")

</div>

Hi, I need to implement the following architecture: my master worker receives some requests (via `HTTP` interface) validates the requests, makes some convertations and puts requests into a queue. Request from the queue is sent to the available worker where computations take place (`do_job`) and results are then sent back to the master worker and then to some output (E.x. `CSV`). I am also thinking about requests prioritization but for now FIFO queue is fine.  
`RemoteChannel` seems to be the closest solution. I can set up two channels

```julia
const requests = Distributed.RemoteChannel(()->Channel{Request}(Inf))
const results = Distributed.RemoteChannel(()->Channel{Result}(Inf))
const worker_pool = WorkerPool(workers())

```

and use the following functions to `put!` requests and schedule them with `remote_do` `do_job`:

```julia
function sched_request(req::Request)
  put!(requests, req)
  @async remote_do(do_job, worker_pool)
  println("task scheduled")
end

@everywhere function do_job()
  req = take!(requests)
  #= 
  do some computations -> res
  =#
  put!(res, results)
  @async remote_do(save_as_csv, master_worker_id)
  println(sent to master worker)
end

```

Is this `RemoteChannel/put!/remote_do` the proper way to deal with such workflow? In this case I have to define `do_job` function on all workers including master worker. However `do_job` is needed only on remote execution workers and `save_as_csv` only on master worker. Maybe there are some recommended examples of such workflow architecture?

---

<div class="post-metadata">

**Author:** ![draftman9](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/draftman9/32/38128_2.png) [@draftman9](https://discourse.julialang.org/u/draftman9)\
**Post date:** [October 14, 2024, 2:34am UTC](https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056/2 "2024-10-14T02:34:00Z")

</div>

Nice idea! I just realized a prototype these days, which works well on the cluster. I have searched all the internet, but I did not find any existing packages about this. I want to develop and release a corresponding package. It will allocate all nodes (cross nodes, all available local and remote CPU cores) on the cluster to execute the tasks in the 'requests` (RemoteChannel). Very relative to your idea. If you (or any viewer) are interested in it, we can cooperate to develop this useful package to fill the hole of julia parallel computation for clusters.

---

<div class="post-metadata">

**Author:** ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Post date:** [October 16, 2024, 3:03am UTC](https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056/3 "2024-10-16T03:03:24Z")

</div>

If you need some collaborator, I could help ✌

---

<div class="post-metadata">

**Author:** ![draftman9](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/draftman9/32/38128_2.png) [@draftman9](https://discourse.julialang.org/u/draftman9)\
**Post date:** [October 16, 2024, 4:44am UTC](https://discourse.julialang.org/t/remotechannels-and-workload-queue-architecture/18056/4 "2024-10-16T04:44:20Z")

</div>

Great! I’ll invite you as soon as I’m done with my duties for the time being.