Track progress of multiple threads

General Usage
1

Hi,
I have the following code


function pb(i, k, lpad, LEN, PBSTR, PBWID,nb)
    perc = i/(LEN);
    lpad = floor(Int,perc * LEN);
    print("\r Processing data chunk ", k," of $nb : [",first(PBSTR,lpad),first(PBWID,LEN-lpad),"] ",trunc(Int,perc*100),"%");
    nothing
end


function progr(;nb::Int=3)
    LEN = 40
    PBSTR = '━'^LEN
    PBWID = ' '^LEN

    print("\e[?25l")
    for k in 1:nb
        for i in 1:LEN
        sleep(0.01*nb)
        pb(i, k, lpad, LEN, PBSTR, PBWID, nb)
        end
        print("\n")
    end
    print("\e[?25h")
end

progr()

I would like to get the 3 progress bars to print progress at the time.
Each task would be running in its own thread.
I have tried to look at packages like ProgressMeter but was not able to figure out how I can do it. Please know that I am not a computer scientist and I am relatively new to Julia. I would like to understand how to do it and understand what is happening.
Thank you for your time.

2

I don’t have much first-hand experience with it myself, but I believe Term.jl would be a good fit for you. Especially look at its documentation regarding progress bars:

https://fedeclaudi.github.io/Term.jl/stable/adv/progressbars/

Adapting your example, it could look like this:

using Term.Progress

function progr(;nb::Int=3)
    LEN = 40

    pbar = ProgressBar()
    jobs = [addjob!(pbar; N=LEN) for _ in 1:nb]

    with(pbar) do
        for k in 1:nb
            for i in 1:LEN
                sleep(0.01*k)
                update!(jobs[k])
            end
        end
    end
end

progr()

Maybe @FedeClaudi will have more insight about this.

3

Thanks. Yes probably both Term.jl and ProgressMeter.jl do it. I wanted to understand how they achieve it because when I read the source code, I am just completely lost and don’t understand what is happening.

4

@gitboy16 we’ve been discussing this on GitHub. Is there in particular you’d like for me to explain about how Term achieves it?

5

I woukd like to k ow how you manage to print all 3 progress bars as the same time while also tracking the progress of each. In my code above, I only manage to print 1 progress bar at a time.
Thank you.

6

The ProgressBar type has a filed jobs which stores all currently active ProgressJobs assigned to the bar. These can be added/removed at any time with addjob!, removejob!.
Every n milliseconds render!(::ProgressBar) gets (see here) called when a progress bar is active and this, in turn, calls render(::ProgressJob) for each job in progress.jobs. Each job is then printed to a separate line.

Each ProgressJob track’s it’s own progress which gets updated with update!(::ProgressJob) and when render! is called produces the visualization to display the current progress.

Note that generally with spawns a separate thread to handle the progress bar updates (see here) separate from the rest of the code.

So I guess the general idea is to use structs to store the information relative to each task and an overarching struct that coordinates the whole thing. Using functions only I don’t think you can achieve what you’re after, or at least not as easily.

Hope this helps, let me know if you have questions!

7

Thank you @FedeClaudi , I guess I am going to have to dive into the code and try to understand what each line of code does. I feel it is going to be a long and painful process… not knowing whether it is going to be succesful are not …

8

I’m sure you can do it! Happy to provide a more in depth explanation.

Are you doing this out of curiousity or with the aim to develop something that Term doesn’t yet offer? Term’s progress bar can do a lot already!

9

As for ProgressMeter.jl, the key is to use the update! function, which is thread-safe as specified in the readme (GitHub - timholy/ProgressMeter.jl: Progress meter for long-running computations)

10

Do you know whether ProgressMeter.jl can display several progress bars at once?

11

I don’t think it can, what I usually do is update the main progress bar with all threads

12

Thank you for your help.
It is more than curiosity. For me it is about learning how to solve a specific problem which I can maybe later use to solve other problems.
I am not planning to develop a package and if a functionnality is missing then (if I have the knowledge) I would try to add it to an existing package.

13

I’ve been working on adding this functionnality to ProgressMeter.jl but it has been paused for some time

You can try this PR with ]add ProgressMeter#a5c5b2c, and I would appreciate feedback

3 Likes
14

Thank you @MarcMush, this is great!
Just a few comments.
1/ in your PR, the very first example does not work for me. I am running Julia 1.9. The rest runs fine.
2/ would you have an example using Threads?
Thank you again.

15

I fixed it, I forgot an i in the example

I don’t know much about threads but it should be working fine
Similarly, use ParallelProgress if you want only one global progressbar or MultipleProgress if each thread should have its own progressbar

1 Like
16

Here are 2 examples with Threads.@threads:

one global progressmeter:

@show Threads.nthreads() # 4
using ProgressMeter
p = ParallelProgress(10)
Threads.@threads for i in 1:10
    sleep(rand())
    next!(p)
end
sleep(0.1)

10 tasks of unknown lengths, spawned with addprogress!:

p = MultipleProgress(Progress(10); count_finishes=true)
Threads.@threads for i in 1:10
    r = rand(10:100)
    addprogress!(p[i], Progress, r; desc="Prog $i ")
    for _ in 1:r
        sleep(0.02)
        next!(p[i])
    end
end
sleep(0.1)
2 Likes
17

Is this PR merged?

18

No, but you can use it with ]add ProgressMeter#a5c5b2c

2 Likes