I’ve got some code where I spawn off a thread to handle SQLite interaction, and then spawn off 4 threads to handle running Agents.jl simulations.
My main function then goes and grabs a simulation description Tuple, put! on a Channel that the worker threads know about, they each take! a simulation tuple and run the sim, and then they put! the data to a channel that talks to the SQLite thread.
The channels each have a depth of 4. once the main thread is done pushing all the Sims, it pushes nothing to the worker channels and waits for the workers. after that it pushes nothing to the SQLite channel, waits for the SQLite thread, and then returns.
Questions:
Is this more or less the right pattern? Are channels compatible with @spawn threads? I think because the sqlite channel is length 0, Julia runtime should block the worker thread and schedule the SQLite thread as soon as a worker completes. Is that right?
it seems to be working… I think. but it seems slower than when serialized. but this is just an impression so far.
I notice that it seems to use only about 3 cpu cores at any given time, though JULIA_NUM_THREADS=4 and I @spawn 4 worker threads, and the SQLite worker is going to spend most of its time blocked on take! waiting for completion of simulations… This seems weird…
here’s my spawning code:
dfchannel = Channel(0)
dbthr = Threads.@spawn sqliteinserter(dfchannel,dbfile)
allthrds = [];
simchannel = Channel(4)
for i in 1:4
t = Threads.@spawn runasim(simchannel,dfchannel)
push!(allthrds,t)
end