Suppose I have multiple processes writing into a single RemoteChannel, and multiple processes reading from it.
Is that an expected “reasonable” use of a RemoteChannel?
If not, is there a better approach?
If yes, is it efficient? E.g., is each message sent directly from writer to reader, or is each message first sent to the process which created the RemoteChannel, and only then to the reader?
Finally, is it possible to take an item from a RemoteChannel if it is not empty, but return immediately if it is currently empty? That is, something like a try_take! function? Calling isready followed by take! doesn’t work since another process may steal the item between calling isready and take!, and locking the RemoteChannel isn’t supported (would also be very inefficient).
Sorry for the noise, but I have the same use-case as you described but I am still struggling to find a suitable implementation pattern. Did you find a good way to do this? Would you mind sharing it?
Edit: In order to maintain state in a stream of incoming data from a variety of different types of sensor, I resorted to have set of RemoteChannels each one containing a circular buffer of length 2.
Also I want to spin up some remote functions with while loops to ingest data from various ports listening to 0MQ and writing the data to channels for the main thread to deal with. I keep hitting race conditions. And fetch() is meaningless as these functions all return nothing.
I’ve just posted a complete explanation here if it’s of any interest.
Not really. I was going to make a serious attempt to port some horrible combination of Python and C++ to Julia to get more performance. The main feature for me was being able to use multiple threads (and also multiple processes on different servers). It turned out that for historical reasons Julia has a strange mixture of multi-processing features and multi threading features that don’t play nicely with each other (e.g. channels and remote channels). Basically, the multi-processing primitives are not thread safe (yikes) so one gets deadlocks (or even crashes). And, as this question shows, they aren’t exactly “feature complete”. It seems like this isn’t a priority for Julia for now compared to other issues; for me this means I’ve given up on it as a serious tool to write high-performance scalable code. I expect this will be fixed in 5-10 years On the bright side, maybe by then they’ll figure out a way to define “contracts” or something so one would be able to implement a type with some guarantee that one covered all the needed functions, instead of having to look up unofficial lists of “to implement T you need to write f, g, h…”. Sigh. To clarify, I like Julia, and I would love to be able to use it instead of the current Python/C++ mess. I just… can’t.
On the bright side, maybe by then they’ll figure out a way to define “contracts” or something so one would be able to implement a type with some guarantee that one covered all the needed functions, instead of having to look up unofficial lists of “to implement T you need to write f, g, h…”. Sigh. To clarify, I like Julia, and I would love to be able to use it instead of the current Python/C++ mess. I just… can’t.