Threading usage patterns: Worker pools for Agents.jl and SQLite.jl

I’ve got some code where I spawn off a thread to handle SQLite interaction, and then spawn off 4 threads to handle running Agents.jl simulations.

My main function then goes and grabs a simulation description Tuple, put! on a Channel that the worker threads know about, they each take! a simulation tuple and run the sim, and then they put! the data to a channel that talks to the SQLite thread.

The channels each have a depth of 4. once the main thread is done pushing all the Sims, it pushes nothing to the worker channels and waits for the workers. after that it pushes nothing to the SQLite channel, waits for the SQLite thread, and then returns.


Is this more or less the right pattern? Are channels compatible with @spawn threads? I think because the sqlite channel is length 0, Julia runtime should block the worker thread and schedule the SQLite thread as soon as a worker completes. Is that right?

it seems to be working… I think. but it seems slower than when serialized. but this is just an impression so far.

I notice that it seems to use only about 3 cpu cores at any given time, though JULIA_NUM_THREADS=4 and I @spawn 4 worker threads, and the SQLite worker is going to spend most of its time blocked on take! waiting for completion of simulations… This seems weird…

here’s my spawning code:

    dfchannel = Channel(0)
    dbthr = Threads.@spawn sqliteinserter(dfchannel,dbfile)
    allthrds = [];
    simchannel = Channel(4)
    for i in 1:4
        t = Threads.@spawn runasim(simchannel,dfchannel)

Having read up on this stuff, it seems that thread migration is not a thing yet… so I’m using only 3 cores due to most likely one of my 4 workers being assigned to the same thread as another one… is there a way to spawn onto specific threads?

For the moment, I’m using JULIA_NUM_THREADS equal to number of real cores + 2, and spawning that many workers, then using htop to limit them to the first set of cores (to avoid oversubscribing cores due to hyperthreading)

In any case, this gets me full CPU usage, and lets the OS handle the slight oversubscription.