# Parallel computing; show progress

**URL:** <https://discourse.julialang.org/t/parallel-computing-show-progress/14033>\
**Category:** General Usage\
**Tags:** question, performance\
**Created:** [August 25, 2018, 9:04am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033 "2018-08-25T09:04:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [August 25, 2018, 9:04am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/1 "2018-08-25T09:04:16Z")

</div>

I have a large number of files to process using pmap() to parallel the calculation. It takes hours to finish. How can I get the percentage of the job?

```julia
# ...
function process_file(file_path)
       ; #process the file
end

file_paths = get_files()
rst = pmap(process_file, file_paths) # block and waited here 

# want to show the progress here 

```

Thanks!

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [August 25, 2018, 10:09am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/2 "2018-08-25T10:09:19Z")

</div>

Look at [GitHub - timholy/ProgressMeter.jl: Progress meter for long-running computations](https://github.com/timholy/ProgressMeter.jl) maybe it’s possible.

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [August 25, 2018, 10:20am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/3 "2018-08-25T10:20:33Z")

</div>

The main problem is that it is blocked and waited when call pmap()

I have used a counter to report the number of the files finished. But still not exactly want I want.

```julia
# ...
function process_file(file_path)
       ; #process the file
        
        global g_n_complete += 1 
        (g_n_complete % 100 == 0 ) && println("worker $myid() finished: ", g_n_complete)
end

file_paths = get_files()
@everywhere global g_n_error
@everywhere global g_n_complete
rst = pmap(process_file, file_paths) # block and waited here 

# want to show the progress here

```

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [August 25, 2018, 10:23am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/4 "2018-08-25T10:23:19Z")

</div>

Thanks for you reply. I have checked the package, but i am not sure how to used it in the parallel computing. When call pmap(), It waits for the return value.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [August 25, 2018, 12:16pm UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/5 "2018-08-25T12:16:16Z")

</div>

You can try the following

```julia
const count = RemoteChannel(()->Channel{Int}(1));
put!(count, 0)

pmap(1:N) do i
    long_computation()
    c = take!(count) + 1
    put!(count, c)
    println("Done with $c jobs")
end

```

---

<div class="post-metadata">

**Author:** ![rickhg12hs](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rickhg12hs/32/5926_2.png) [@rickhg12hs](https://discourse.julialang.org/u/rickhg12hs)\
**Post date:** [August 27, 2018, 11:25am UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/6 "2018-08-27T11:25:22Z")

</div>

Just wondering … Rather than thinking of this as a parallel computing issue, could it be approached by considering a new type of collection that `pmap` iterates through? To `pmap` the collection would be the same, but it would have extra features. For example, this new collection might have some feature that provides a window into how far into the collection it has iterated, etc. Maybe a collection with an iterator that can publish/subscribe or be queried?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [August 27, 2018, 2:33pm UTC](https://discourse.julialang.org/t/parallel-computing-show-progress/14033/7 "2018-08-27T14:33:20Z")

</div>

It sounds like you are talking about a `RemoteChannel`  
`put!` and `take!` serve as publishing and subscribing mechanisms.