Track progress of multiple threads

Hi,
I have the following code


function pb(i, k, lpad, LEN, PBSTR, PBWID,nb)
    perc = i/(LEN);
    lpad = floor(Int,perc * LEN);
    print("\r Processing data chunk ", k," of $nb : [",first(PBSTR,lpad),first(PBWID,LEN-lpad),"] ",trunc(Int,perc*100),"%");
    nothing
end


function progr(;nb::Int=3)
    LEN = 40
    PBSTR = '━'^LEN
    PBWID = ' '^LEN

    print("\e[?25l")
    for k in 1:nb
        for i in 1:LEN
        sleep(0.01*nb)
        pb(i, k, lpad, LEN, PBSTR, PBWID, nb)
        end
        print("\n")
    end
    print("\e[?25h")
end

progr()

I would like to get the 3 progress bars to print progress at the time.
Each task would be running in its own thread.
I have tried to look at packages like ProgressMeter but was not able to figure out how I can do it. Please know that I am not a computer scientist and I am relatively new to Julia. I would like to understand how to do it and understand what is happening.
Thank you for your time.

I don’t have much first-hand experience with it myself, but I believe Term.jl would be a good fit for you. Especially look at its documentation regarding progress bars:

https://fedeclaudi.github.io/Term.jl/stable/adv/progressbars/

Adapting your example, it could look like this:

using Term.Progress

function progr(;nb::Int=3)
    LEN = 40

    pbar = ProgressBar()
    jobs = [addjob!(pbar; N=LEN) for _ in 1:nb]

    with(pbar) do
        for k in 1:nb
            for i in 1:LEN
                sleep(0.01*k)
                update!(jobs[k])
            end
        end
    end
end

progr()


Maybe @FedeClaudi will have more insight about this.

Thanks. Yes probably both Term.jl and ProgressMeter.jl do it. I wanted to understand how they achieve it because when I read the source code, I am just completely lost and don’t understand what is happening.

@gitboy16 we’ve been discussing this on GitHub. Is there in particular you’d like for me to explain about how Term achieves it?

I woukd like to k ow how you manage to print all 3 progress bars as the same time while also tracking the progress of each. In my code above, I only manage to print 1 progress bar at a time.
Thank you.

The ProgressBar type has a filed jobs which stores all currently active ProgressJobs assigned to the bar. These can be added/removed at any time with addjob!, removejob!.
Every n milliseconds render!(::ProgressBar) gets (see here) called when a progress bar is active and this, in turn, calls render(::ProgressJob) for each job in progress.jobs. Each job is then printed to a separate line.

Each ProgressJob track’s it’s own progress which gets updated with update!(::ProgressJob) and when render! is called produces the visualization to display the current progress.

Note that generally with spawns a separate thread to handle the progress bar updates (see here) separate from the rest of the code.

So I guess the general idea is to use structs to store the information relative to each task and an overarching struct that coordinates the whole thing. Using functions only I don’t think you can achieve what you’re after, or at least not as easily.

Hope this helps, let me know if you have questions!

Thank you @FedeClaudi , I guess I am going to have to dive into the code and try to understand what each line of code does. I feel it is going to be a long and painful process… not knowing whether it is going to be succesful are not …

I’m sure you can do it! Happy to provide a more in depth explanation.

Are you doing this out of curiousity or with the aim to develop something that Term doesn’t yet offer? Term’s progress bar can do a lot already!

As for ProgressMeter.jl, the key is to use the update! function, which is thread-safe as specified in the readme (GitHub - timholy/ProgressMeter.jl: Progress meter for long-running computations)

Do you know whether ProgressMeter.jl can display several progress bars at once?

I don’t think it can, what I usually do is update the main progress bar with all threads

Thank you for your help.
It is more than curiosity. For me it is about learning how to solve a specific problem which I can maybe later use to solve other problems.
I am not planning to develop a package and if a functionnality is missing then (if I have the knowledge) I would try to add it to an existing package.