# Keeping threads busy with 1.3 multiprocessing

**URL:** <https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253>\
**Category:** Performance\
**Created:** [December 13, 2019, 9:52pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253 "2019-12-13T21:52:09Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![babazaroni](https://avatars.discourse-cdn.com/v4/letter/b/a88e4f/32.png) [@babazaroni](https://discourse.julialang.org/u/babazaroni)\
**Post date:** [December 13, 2019, 9:52pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/1 "2019-12-13T21:52:09Z")

</div>

I’m filtering 185 large historical monthly csv files and am allocating a thread to process a month. When I make an initial 185 thread requests, the system starts out allocating all available threads (8 on my machine) but then slowly winds down till the end where it’s only using 1 thread. This method take 3.5 hours.

When I limit the number of pending requests to 8, the system is kept busy more often, but there are times when the active thread count drops, then ramps back to 7, the limit I’ve set for Julia. This method takes 2.5 hours, an hour faster.

I am using Ubuntu 18.04 on intel machine.

Here is the code for making bulk 185 thread requests:

```julia
println(Dates.format(now(),"HH:MM"))

@sync for filename in readdir(inpath)
    
    Threads.@spawn process_month(filename)
    
end

println(Dates.format(now(),"HH:MM"))

```

Here is the code limiting the pending requests to 8 threads:

```julia
println(Dates.format(now(),"HH:MM"))

NUM_THREADS = 8

threadflags = [0 for x in 1:NUM_THREADS]

@sync for filename in readdir(inpath)
    
    while(isnothing(findfirst(isequal(0),threadflags)))
        sleep(1)
    end
    
    i = findfirst(isequal(0),threadflags)
    
    threadflags[i] = 1
    Threads.@spawn process_month(filename,i)
    
end

println(Dates.format(now(),"HH:MM"))

```

Any tips for keeping more threads busy more often?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 13, 2019, 9:58pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/2 "2019-12-13T21:58:38Z")

</div>

Are you sure you have fully optimized the worker operation? If not, working on the multi-threading will probably be harder to get the same potential benefits.

---

<div class="post-metadata">

**Author:** ![babazaroni](https://avatars.discourse-cdn.com/v4/letter/b/a88e4f/32.png) [@babazaroni](https://discourse.julialang.org/u/babazaroni)\
**Post date:** [December 13, 2019, 10:08pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/3 "2019-12-13T22:08:27Z")

</div>

The worker is simple, so I don’t know how to optimize further.

```
    df = readtable(f,eltypes = types_filter)
    df = df[findall(in(filterlist),df[!, :symbol]),:]

    append!(df_data,df)

```

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [December 13, 2019, 10:12pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/4 "2019-12-13T22:12:54Z")

</div>

Orthogonal issue (and sorry to take over your thread), but wouldn’t `readtable` in multiple threads be really slow because of harddrive jumping around to get the data?

---

<div class="post-metadata">

**Author:** ![babazaroni](https://avatars.discourse-cdn.com/v4/letter/b/a88e4f/32.png) [@babazaroni](https://discourse.julialang.org/u/babazaroni)\
**Post date:** [December 13, 2019, 10:19pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/5 "2019-12-13T22:19:11Z")

</div>

Interesting idea. Read and write the data from a single thread. Do the filtering in separate thread.

I am using SSD, instead of rotating disk if that matters.

Still, I wonder why the big difference in performance between these two methods. From watching the 2019 JuliaCon video about the new 1.3 processing, it was clear you can make many requests beyond your physical thread count and the scheduler would keep the machine busy till all thread requests were satisfied. This happens for about the first few minutes, but then degrades. Often only a single thread is operating on the data.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 13, 2019, 10:45pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/6 "2019-12-13T22:45:12Z")

</div>

For an SSD, my guess is that multiprocessed reads would be better, but benchmarking is always smart.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 14, 2019, 12:32am UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/7 "2019-12-14T00:32:25Z")

</div>

Did you measure time it took to run each `process_month`? Is it possible that there are a few files that is much larger than others?

Aside: Is `df_data` in `append!(df_data,df)` shared across threads? If so, I think your code as-is would corrupt the data.

Aside^2: I think the best strategy for using threads is to use map/reduce/filter framework. For example, using [`Transducers.reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce), your code is simply something like `reduce(append!!, Map(process_month′), readdir(inpath); init=Empty(DataFrame), basesize=1)` provided that `process_month′` is a function that returns a DataFrame. Demo:

```julia
julia> using Transducers, BangBang, DataFrames

julia> reduce(append!!, Map(x -> DataFrame(a=[x])), 1:2; init=Empty(DataFrame), basesize=1)
2×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │
│ 2 │ 2 │

```

---

<div class="post-metadata">

**Author:** ![babazaroni](https://avatars.discourse-cdn.com/v4/letter/b/a88e4f/32.png) [@babazaroni](https://discourse.julialang.org/u/babazaroni)\
**Post date:** [December 15, 2019, 2:18am UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/8 "2019-12-15T02:18:40Z")

</div>

I’ve been doing more experiments in trying to keep more threads running.

What is working for me is to not give up threads until all work is completed. In the previous attempts described above, I would give up a thread after a month was processed, then request a thread for the next month. This results in inconsistent thread count ranging from the max to a single thread.

The way that works is to start a thread that requests months to process from a channel. I’ll start 6 such threads, leaving 2 for the system. Any more than that and I start getting inconsistent thread behavior.

So I believe the scheduler is not working as advertised. Multiprocessing in 1.3 is experimental, so I’ll revisit this when there are improvements.

---

<div class="post-metadata">

**Author:** ![Ratingulate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ratingulate/32/9242_2.png) [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)\
**Post date:** [December 15, 2019, 6:52pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/9 "2019-12-15T18:52:49Z")

</div>

If you think this is a bug, would you consider opening an issue?

---

<div class="post-metadata">

**Author:** ![bocc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bocc/32/7546_2.png) [@bocc](https://discourse.julialang.org/u/bocc)\
**Post date:** [December 15, 2019, 7:21pm UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/10 "2019-12-15T19:21:40Z")

</div>

As **tkf** said, mutating a variable concurrently from multiple threads is problematic, you should protect your `tf_data` with [some sort of lock](https://docs.julialang.org/en/v1/manual/faq/#faq-async-io-1) to avoid intermixing. (Sorry if you’re already doing that.)

---

<div class="post-metadata">

**Author:** ![babazaroni](https://avatars.discourse-cdn.com/v4/letter/b/a88e4f/32.png) [@babazaroni](https://discourse.julialang.org/u/babazaroni)\
**Post date:** [December 16, 2019, 12:37am UTC](https://discourse.julialang.org/t/keeping-threads-busy-with-1-3-multiprocessing/32253/11 "2019-12-16T00:37:23Z")

</div>

There is no corruption issue here. This thread is about inconsistent thread scheduling.