got dual xeon 2699 v3 so 36 cores. If I run threaded code each of which generates a stream of data values, from say a temperature sensor, on 10 of them and in the threaded code each writes it’s thread number and a data value to a single queue which is consumed by another thread on another core, are there any issues?
sorry I thought it was clear. I mean the queue in
https://juliacollections.github.io/DataStructures.jl/latest/stack_and_queue/
I would be surprised if that worked without races. AFAIK multi threading is not supported in DataStructures.jl and the data structures are not lock free. You would have to implement your own locking scheme.
1 Like
really how is that possible? I would have thought that the act of using a queue negates any possibility of a race condition. The thread just puts to the next slot in the queue. Remember the queue is running on different thread.
How do you send the info between threads? If the queue is private to a single thread that manages it, then any race problem will not happen at the queue, but instead they will depend on how this queue-manager thread gets the information from the other threads.
1 Like
the queue will be opened by the threads
You are really not giving any sufficient amount of detail for us to give you an answer.
If you meant that every thread will have shared access to the queue then yes, race conditions can happen. One thread may be interrupted in the middle of adding a new item, and when the execution returns to the thread it can overwrite the position that was unused before.
1 Like
I am so sorry @Henrique_Becker you are completely right I haven’t been too clear on this at all. I DID mean each thread had access to the same queue and thought that there would be NO race condition. I suppose it’s back to the drawing board. My concept was that I would allocate a process to a thread which would only gather data and place it on a shared queue running on a separate thread. I thought that basic queue functionality would kick in that the “data” thread could NOT be interrupted by anything (thread isolation) so the push to the queue would always succeed. So I would start up three threads, two would gather data. The third thread would run the queue. The two data threads would open the queue and write to it. That was how I saw this working. I used to do the same thing on a HP system using a shared databus called $receive and micromessages.
Thank you all for your help
As others have mentioned, Channel is the go-to solution if you need a concurrent queue; i.e., a thread-safe FIFO container. Also, note that you need to assume that the data structure APIs are not thread-safe unless it is explicitly documented as such. If you really want to use non-thread-safe data structure like the queue in DataStructures.jl, put a lock (ReentrantLock) around it. However, you won’t get extra functionality like waitable take! implemented by Channel. If you are sure that Channel won’t work for you, have a look at various concurrent queue implementations in https://github.com/JuliaConcurrent/ConcurrentCollections.jl (note that this package is pretty new, not tested well in the wild, and requires Julia 1.7)
5 Likes
as @tkf said, I’m pretty sure what you want is a Channel. It’s designed to do exactly what you want. Your “producer” threads write to it, and a consumer thread reads from it. You may want to give it a buffer of length approximately equal to the number of producer threads, or the number of threads x2 for example. This means the producers will not block (provided the consumer consumes at an appropriate pace).
1 Like
@dlakelan @tkf @Henrique_Becker @fredrikekre Thank you all for taking the time to set me straight. I misread the documentation for DataStructures.jl . I will spend some time researching Channel. Thank you all for taking the time on a Saturday to help a newbie who clearly can’t read 
1 Like
Can we really be certain that channels are threadsafe? The docs do not seem to state that anywhere. Furthermore, in the docs channels are only featured in relation to asynchronous programming and not in true multithreading.
Yes, agree. Looking at the Channel implementation it does indeed look thread-safe, e.g. with the use of locks for each put! and take! calls.
The Julia docs could certainly need an upgrade here though.
Watching this video as a result of all your help seems to me that it’s EXACTLY what I would like to do ( threaded workers) and the presenter is doing a marvelous job that even someone like me can grasp.