Some feedback while waiting for pmap

I am running a long calculation distributed on several workers and several computers:
output = pmap( i->iter(i,parameters), 1:1000 )

While waiting I would like to know which i is being processed.
How can I get this simple info on master? (without writing files)

1 Like

The poor man’s solution (aka my solution) was print statements inside the function being called.

But how can I get rid of the “From worker:” part of the printed messages?
Also I would like to see less than a thousand lines.

No idea. I think there is some work being done on ProgressMeter.jl to add functionality for pmap

Thank you very much for pointing me to ProgressMeter.jl, it is definitely a useful package.

But unfortunately, pmap seems to be out of its focus:

@showprogress 1 output=pmap( i->iter(i,parameters), 1:1000 )

ERROR: LoadError: LoadError: ArgumentError: Final argument to @showprogress must be a for loop or comprehension.

You could set up a RemoteChannel and have workers query the channel before printing. The master node could then write a value to the channel indicating that you want a worker to print some info.

Thank you for the advice.
As I have never used remote channels before,
any short code example would be acknowledged.

using Distributed
const channel = RemoteChannel(()->Channel{Int}(1))

@async pmap(1:20) do i
    isready(channel) && println(i)

Then to activate printing put!(channel, 1) and to deactivate printing take!(channel).

Clever code, thank you very much.

My remaining problem is the same as was with the “poor man’s solution”.
When getting the feedback info using println I can see the results,
but also the “From worker:” extra messages and 1000 separate lines.
If I try to replace println by the function print, then nothing is printed.

Well, you could have the worker put it’s latest iteration index and the master fetching it and doing the printing