Hey there,
I would like to suggest a nice idea to improve parallel programming in Julia.
I think it would be nice to be able to build a pipeline of functions operating on a stream of data by simply programming the functions given a single input batch, and assigning workers (procs) to each stage in the process to balance the “production line”. To achieve this, data must be produced/generated to queues/buffers between functions and consumed by the next function inline if some worker is free downstream. In other words, the program should be able to generate tasks using multiple procs and consume them by any proc available in the downstream stage. I tried naive implementations using pmap, generate/produce and PyGen but my black magic was not strong enough, so I thought of asking the great Magicians here for help!