I have used Julia Channels in real-time processing of data in a pipeline fashion. Each Channel passes data to the next processing step. A simplified example is the code below, one channel for generating random numbers and another to generate the averages of these numbers. This 2nd Channel is consumer for the 1st Channel data but also a producer for the following stage. I hope the code gives the essence of what I am after.
function randomsrc() function cr(ch) while true put!(ch, randn()) end end return Channel(cr, ctype=Float64, csize=1024) end function avsrc(c) sum = 0.0 count = 0 function av(ch) for x in c sum += x count += 1 put!(ch, sum/count) end end return Channel(av, ctype=Float64, csize=1024) end randch = randomsrc() avch = avsrc(randch) for (i, a) in enumerate(avch) if i % 100000 == 0 @show(i, a) end end
This works fine but I have performance issues when the data rate increases and/or when some of the intermediate stages start to do more with the incoming data, e.g. plotting it.
It seems from what I understood in the documentation, Channels are single threaded. Is there a parallel construct that helps distribute the processing to different threads on different cores to achieve higher throughput? I like the ease of use of the Channels and how intuitive using them for a pipeline-like problems, or chaining producers/consumers. Any pointers to similar constructs that can help me?