Producer consumer pattern in julia

Specific Domains Julia at Scale
1

I’m implementing a producer consumer pattern in julia. Each producer will be pushing things on a queue and each consumer will be poping them off. I’d like each producer and each consumer to run in a different execution thread (or process). Any thoughts on the best way to implement this? Below are my current thoughts (not yet tried):

  1. It seems that the thread infrastructure in v0.6 doesn’t easily support this.
  2. It seems it’s easiest to use SharedArray’s for the queue with processes for the producers and consumers.
  3. Do I need to use synchronization with SharedArray’s? I assume push! and pull! are not atomic. I see that julia has synchronization primitives but they seem to be for threads rather than proceses.
2

I could be very wrong but I think that Remote Channels are intended to handle this type of thing.

3

Keep in mind that SharedArray only works on the same server. If you need to parallelize the work across multiple servers you will pay the price of transmitting data.

I recently did something similar utilizing ZMQ.jl PUSH/PULL sockets. The producer pushes requests and the consumers receive them in a round robin fashion. That works quite well and it’s just several lines of code to hook things up.