Displaying Parallel Progress Bars


#1

I need to run many simulations that will each take several hours to complete. To speed up the process, I’m going to run several of these simulations in parallel. The trouble is, I’m not very patient, and so I want some sort of progress indicator for each simulation.

I am aware of PmapProgressMeter.jl, but it isn’t quite what I had in mind. I would prefer a series of progress bars that shows the progress for the currently running simulation on each processor, as well as the number of simulations remaining to start.

It would look something like this:

5 Tasks remaining to start
--------------------------
Worker 2:  28%|████████████                       |  ETA: 0:44:11
  .
  .
  .
Worker 5:  65%|█████████████████                  |  ETA: 1:26:17

Does anyone know of a package that can do this?


#2

If you use Juno you can use its progress bar in a nested fashion.


#3

Thanks for the suggestion, but I’m working over ssh so I need a terminal based solution.


#4

There’s a WIP PR for this kind of API to be in Base, but I’m not sure the solution you’re looking for exists quite yet:


#5

In case this helps anyone else, I’ve hacked together a working version of this functionality using ProgressMeter.jl as a base. I’ve been using it for a while now, and it seems to work well enough. If anyone has suggestions, I would be happy to incorporate them. You can find the package here:

To use it, add some worker processes and define a work function:

addprocs(4)

@everywhere using ProgressMeter

@everywhere function test_function(x, p=nothing)
    @info("Running with value $x")

    @parallelprogress p "optional label " for i in 1:x
        sleep(0.1)
    end

    @info("Done")
    return x
end

Then it can either be run directly on the master process, or farmed out to the worker processes:

julia> test_function(100)
INFO: Running with value 100
optional label 100%|████████████████████████████████████| Time: 0:00:10
INFO: Done
100

julia> ProgressMeter.pmap(test_function, [50, 50, 100, 25, 10, 50])
2 tasks remaining to start
-------------------------------
optional label  40%|██████████████                      |  ETA: 0:00:04
optional label  80%|█████████████████████████████       |  ETA: 0:00:01
optional label  40%|██████████████                      |  ETA: 0:00:04
optional label  20%|███████                             |  ETA: 0:00:10

The @info macros allow you to print diagnostic information while you are still developing, and then easily run the function in parallel once everything is working.


Does anyone have a progress bar for pmap?