# ProgressMeter generates lots of allocations and overhead when multi-threaded (even if disabled)

**URL:** <https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696>\
**Category:** General Usage\
**Created:** [June 28, 2021, 1:26pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696 "2021-06-28T13:26:53Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 1:26pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/1 "2021-06-28T13:26:53Z")

</div>

Maybe I should post an issue on the package site, but lets go. Given this simple threaded loop:

```julia
julia> using ProgressMeter, BenchmarkTools

julia> function loop(n)
         s = zeros(Threads.nthreads())
         Threads.@threads for i in 1:n
           s[Threads.threadid()] += sin(i)
         end
         sum(s)
       end
loop (generic function with 1 method)

julia> @btime loop(1000000)
  4.401 ms (22 allocations: 1.97 KiB)
-0.1171095240981539

```

Now if I add a progress meter, even if it is disabled, I get:

```julia
julia> function loop(n)
         p = Progress(n,dt=0.1,enabled=false)
         s = zeros(Threads.nthreads())
         Threads.@threads for i in 1:n
           s[Threads.threadid()] += sin(i)
           next!(p)
         end
         sum(s)
       end
loop (generic function with 1 method)

julia> @btime loop(1000000)
  435.372 ms (369003 allocations: 11.26 MiB)
-0.1171095240981539

```

Thus, I get lots of allocations, and the overhead is also very large, considering that with it disabled I imagine that it should only return from `next!` without doing nothing.

I can even add this simple `mynext!` function to check if the progress meter is enabled, to get a much better result:

```julia
julia> function mynext!(p::Progress)
         if p.enabled
           next!(p)
         end
         nothing
       end
mynext! (generic function with 1 method)

julia> function loop(n)
         p = Progress(n,dt=0.1,enabled=false)
         s = zeros(Threads.nthreads())
         Threads.@threads for i in 1:n
           s[Threads.threadid()] += sin(i)
           mynext!(p)
         end
         sum(s)
       end
loop (generic function with 1 method)

julia> @btime loop(1000000)
  5.521 ms (31 allocations: 2.50 KiB)
-0.1171095240981539

```

I am doing something wrong, or should I just post an issue?

Thanks.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 1:53pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/2 "2021-06-28T13:53:02Z")

</div>

I added a simple pull request that fixes that behavior: [return immediately from next! if not enabled by lmiq · Pull Request #213 · timholy/ProgressMeter.jl · GitHub](https://github.com/timholy/ProgressMeter.jl/pull/213)

Perhaps it is not what the developers would propose, but there it is.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [June 28, 2021, 2:19pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/3 "2021-06-28T14:19:39Z")

</div>

One guess would be that a single progress meter should not be shared across threads.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 2:27pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/4 "2021-06-28T14:27:02Z")

</div>

Yes, you can, it is thread-safe. Basic example from the docs:

```julia
using ProgressMeter
p = Progress(10)
Threads.@threads for i in 1:10
    sleep(2*rand())
    next!(p)
end

```

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [June 28, 2021, 2:30pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/5 "2021-06-28T14:30:07Z")

</div>

I know nothing about internals of ProgressMeter, but I can guess, that in order for `next` to be thread safe operation, it should lock progressmeter. And each lock generates a pause in all calculations. In loop with 1\_000\_000 iterations you get 1\_000\_000 locks and it’ll generate a lot of overhead (and probably allocations).

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 2:31pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/6 "2021-06-28T14:31:55Z")

</div>

All your suppositions are correct :-). Yet I think that when it is disabled it should avoid all that and just return.

([it does that](https://github.com/timholy/ProgressMeter.jl/blob/e1c7c7127848ee270b6770e03971d326afad2b85/src/ProgressMeter.jl#L407) one step down from the `next!` function, but apparently should be done earlier).

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [June 28, 2021, 2:36pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/7 "2021-06-28T14:36:11Z")

</div>

> [@lmiq](#):
>
> `enabled`

you’re just disabling the output, but it still keeping track of your progress in case you want to see it later

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 2:41pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/8 "2021-06-28T14:41:07Z")

</div>

I see…

Then what I would want is a feature, which would be disabling it completely. (I can live with just creating my function that wraps `next!`, then).

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 2:42pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/10 "2021-06-28T14:42:43Z")

</div>

This loop is buried inside a package and should be controlled by a user option.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [June 28, 2021, 2:44pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/11 "2021-06-28T14:44:27Z")

</div>

use your flag then:

```julia
p = Progress(n,dt=0.1,enabled=do_progress)
...
...
do_progress && next!(p)

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 28, 2021, 2:49pm UTC](https://discourse.julialang.org/t/progressmeter-generates-lots-of-allocations-and-overhead-when-multi-threaded-even-if-disabled/63696/12 "2021-06-28T14:49:10Z")

</div>

Well thought 🙂