Displaying Parallel Progress Bars

General Usage
1

I need to run many simulations that will each take several hours to complete. To speed up the process, I’m going to run several of these simulations in parallel. The trouble is, I’m not very patient, and so I want some sort of progress indicator for each simulation.

I am aware of PmapProgressMeter.jl, but it isn’t quite what I had in mind. I would prefer a series of progress bars that shows the progress for the currently running simulation on each processor, as well as the number of simulations remaining to start.

It would look something like this:

5 Tasks remaining to start
--------------------------
Worker 2:  28%|████████████                       |  ETA: 0:44:11
  .
  .
  .
Worker 5:  65%|█████████████████                  |  ETA: 1:26:17

Does anyone know of a package that can do this?

2 Likes
2

If you use Juno you can use its progress bar in a nested fashion.

3

Thanks for the suggestion, but I’m working over ssh so I need a terminal based solution.

4

There’s a WIP PR for this kind of API to be in Base, but I’m not sure the solution you’re looking for exists quite yet:

https://github.com/JuliaLang/Juleps/pull/30#issuecomment-301248423

5

In case this helps anyone else, I’ve hacked together a working version of this functionality using ProgressMeter.jl as a base. I’ve been using it for a while now, and it seems to work well enough. If anyone has suggestions, I would be happy to incorporate them. You can find the package here:

https://github.com/adamslc/ProgressMeter.jl

To use it, add some worker processes and define a work function:

addprocs(4)

@everywhere using ProgressMeter

@everywhere function test_function(x, p=nothing)
    @info("Running with value $x")

    @parallelprogress p "optional label " for i in 1:x
        sleep(0.1)
    end

    @info("Done")
    return x
end

Then it can either be run directly on the master process, or farmed out to the worker processes:

julia> test_function(100)
INFO: Running with value 100
optional label 100%|████████████████████████████████████| Time: 0:00:10
INFO: Done
100

julia> ProgressMeter.pmap(test_function, [50, 50, 100, 25, 10, 50])
2 tasks remaining to start
-------------------------------
optional label  40%|██████████████                      |  ETA: 0:00:04
optional label  80%|█████████████████████████████       |  ETA: 0:00:01
optional label  40%|██████████████                      |  ETA: 0:00:04
optional label  20%|███████                             |  ETA: 0:00:10

The @info macros allow you to print diagnostic information while you are still developing, and then easily run the function in parallel once everything is working.

4 Likes
6

It’s been a while since this post. Is there now a registered package that provides this kind of parallel progress bar setup for pmap and @distributed processes?

8 Likes
7

I’ve been looking for something like this (multiple/nested progress) myself. ProgressLoggers kinda does it, but not cleanly and only with TerminalLoggers, it seems.

8

Back when Juno was still maintained, I wrote a simple package for following progress of Distributed workers, which could be displayed inside Juno plot pane. In combination with TerminalLoggers it can show a basic display of textual progress bars.
https://github.com/yha/ObservablePmap.jl
Unfortunately, it doesn’t work in vscode plot pane, but it can work inside a Blink window, like this:

using Distributed
using Blink: Window, body!
addprocs(...)

@everywhere using ObservablePmap
@everywhere using ProgressLogging
@everywhere using TerminalLoggers

summ, task = ologpmap( 'a':'e', 2:2:10; logger_f=TerminalLogger ) do c, x
    @withprogress name="Processing '$c'" for i=1:x
        sleep(rand())
        @logprogress i/x
    end
    @info "finished '$c'."
    x
end

html = map(x -> HTML("<pre>$x</pre>"), summ)

w = Window()
body!(w, html)

The downside is that you need a dedicated window for showing the progress. Also, on my Windows machine that <pre> tag doesn’t seem to respected in the Blink window for some reason - it uses a non-monospace font, so the progress bars don’t quite look right.

1 Like