Some feedback while waiting for pmap


#1

I am running a long calculation distributed on several workers and several computers:
output = pmap( i->iter(i,parameters), 1:1000 )

While waiting I would like to know which i is being processed.
How can I get this simple info on master? (without writing files)


#2

The poor man’s solution (aka my solution) was print statements inside the function being called.


#3

Thanks.
But how can I get rid of the “From worker:” part of the printed messages?
Also I would like to see less than a thousand lines.


#4

No idea. I think there is some work being done on ProgressMeter.jl to add functionality for pmap


#5

Thank you very much for pointing me to ProgressMeter.jl, it is definitely a useful package.

But unfortunately, pmap seems to be out of its focus:

@showprogress 1 output=pmap( i->iter(i,parameters), 1:1000 )

ERROR: LoadError: LoadError: ArgumentError: Final argument to @showprogress must be a for loop or comprehension.


#6

You could set up a RemoteChannel and have workers query the channel before printing. The master node could then write a value to the channel indicating that you want a worker to print some info.


#7

Thank you for the advice.
As I have never used remote channels before,
any short code example would be acknowledged.


#8
using Distributed
addprocs(2)
const channel = RemoteChannel(()->Channel{Int}(1))

@async pmap(1:20) do i
    isready(channel) && println(i)
    sleep(3)
end

Then to activate printing put!(channel, 1) and to deactivate printing take!(channel).


#9

Clever code, thank you very much.

My remaining problem is the same as was with the “poor man’s solution”.
When getting the feedback info using println I can see the results,
but also the “From worker:” extra messages and 1000 separate lines.
If I try to replace println by the function print, then nothing is printed.


#10

Well, you could have the worker put it’s latest iteration index and the master fetching it and doing the printing