turtle
September 14, 2018, 6:27pm
1
I am running a long calculation distributed on several workers and several computers:
output = pmap( i->iter(i,parameters), 1:1000 )
While waiting I would like to know which i
is being processed.
How can I get this simple info on master? (without writing files)
1 Like
tbeason
September 14, 2018, 8:02pm
2
The poor man’s solution (aka my solution) was print statements inside the function being called.
turtle
September 14, 2018, 9:30pm
3
Thanks.
But how can I get rid of the “From worker:” part of the printed messages?
Also I would like to see less than a thousand lines.
tbeason
September 14, 2018, 10:14pm
4
No idea. I think there is some work being done on ProgressMeter.jl to add functionality for pmap
turtle
September 15, 2018, 8:29am
5
Thank you very much for pointing me to ProgressMeter.jl
, it is definitely a useful package.
But unfortunately, pmap
seems to be out of its focus:
@showprogress 1 output=pmap( i->iter(i,parameters), 1:1000 )
ERROR: LoadError: LoadError: ArgumentError: Final argument to @showprogress must be a for loop or comprehension.
You could set up a RemoteChannel and have workers query the channel before printing. The master node could then write a value to the channel indicating that you want a worker to print some info.
turtle
September 16, 2018, 7:19am
7
Thank you for the advice.
As I have never used remote channels before,
any short code example would be acknowledged.
using Distributed
addprocs(2)
const channel = RemoteChannel(()->Channel{Int}(1))
@async pmap(1:20) do i
isready(channel) && println(i)
sleep(3)
end
Then to activate printing put!(channel, 1)
and to deactivate printing take!(channel)
.
turtle
September 16, 2018, 10:13am
9
Clever code, thank you very much.
My remaining problem is the same as was with the “poor man’s solution”.
When getting the feedback info using println
I can see the results,
but also the “From worker:” extra messages and 1000 separate lines.
If I try to replace println
by the function print
, then nothing is printed.
Well, you could have the worker put it’s latest iteration index and the master fetching it and doing the printing