# Threading usage patterns: Worker pools for Agents.jl and SQLite.jl

**URL:** <https://discourse.julialang.org/t/threading-usage-patterns-worker-pools-for-agents-jl-and-sqlite-jl/44232>\
**Category:** General Usage\
**Tags:** multithreading, sqlite, agents\
**Created:** [August 4, 2020, 4:48am UTC](https://discourse.julialang.org/t/threading-usage-patterns-worker-pools-for-agents-jl-and-sqlite-jl/44232 "2020-08-04T04:48:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2020, 4:48am UTC](https://discourse.julialang.org/t/threading-usage-patterns-worker-pools-for-agents-jl-and-sqlite-jl/44232/1 "2020-08-04T04:48:07Z")

</div>

I’ve got some code where I spawn off a thread to handle SQLite interaction, and then spawn off 4 threads to handle running Agents.jl simulations.

My main function then goes and grabs a simulation description Tuple, put! on a Channel that the worker threads know about, they each take! a simulation tuple and run the sim, and then they put! the data to a channel that talks to the SQLite thread.

The channels each have a depth of 4. once the main thread is done pushing all the Sims, it pushes `nothing` to the worker channels and waits for the workers. after that it pushes nothing to the SQLite channel, waits for the SQLite thread, and then returns.

## Questions:

Is this more or less the right pattern? Are channels compatible with `@spawn` threads? I think because the sqlite channel is length 0, Julia runtime should block the worker thread and schedule the SQLite thread as soon as a worker completes. Is that right?

it seems to be working… I think. but it seems slower than when serialized. but this is just an impression so far.

I notice that it seems to use only about 3 cpu cores at any given time, though JULIA\_NUM\_THREADS=4 and I `@spawn` 4 worker threads, and the SQLite worker is going to spend most of its time blocked on take! waiting for completion of simulations… This seems weird…

here’s my spawning code:

```julia-auto
    dfchannel = Channel(0)
    dbthr = Threads.@spawn sqliteinserter(dfchannel,dbfile)
    allthrds = [];
    simchannel = Channel(4)
    
    for i in 1:4
        t = Threads.@spawn runasim(simchannel,dfchannel)
        push!(allthrds,t)
    end

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2020, 2:02pm UTC](https://discourse.julialang.org/t/threading-usage-patterns-worker-pools-for-agents-jl-and-sqlite-jl/44232/2 "2020-08-04T14:02:45Z")

</div>

Having read up on this stuff, it seems that thread migration is not a thing yet… so I’m using only 3 cores due to most likely one of my 4 workers being assigned to the same thread as another one… is there a way to spawn onto specific threads?

For the moment, I’m using JULIA\_NUM\_THREADS equal to number of real cores + 2, and spawning that many workers, then using htop to limit them to the first set of cores (to avoid oversubscribing cores due to hyperthreading)

In any case, this gets me full CPU usage, and lets the OS handle the slight oversubscription.