MPI and println inside imported modules

General Usage
, , , ,
1

Dear Juliawers,
I’m using optimizing a function in julia which is parallelized using MPI.
Each time I have to print something, I check if the rank is zero to avoid printing the text as many times as the number of processors.
To optimize I’m using the Optim module.
To print in stdout the progress of the optimization as it goes, I pass to the optimize function the show_trace and show_every variables.

However, if I’m running on several processors, the status of the minimization is printed many times. How can I override the print statements in the module imported? Is there any way to do so without modifying the source code of Optim?

Example:


V(r, θ) = a (r - r0)² + b cos(θ - θ0)

Parameters:
   r0         = 2.0
   theta0     = 0.0
   a          = 6.0
   b          = 1.0
   n_configs  = 10000
   n_layers   = 1

[ Info: Optimizing the free energy using 4 processor(s)
x0 = [1.0, 0.0, 1.0, 2.0, 0.0, 0.0, 1.0, 0.0, 1.0, 2.0, 0.0]
Iter     Function value   Gradient norm 
Iter     Function value   Gradient norm 
Iter     Function value   Gradient norm 
Iter     Function value   Gradient norm 
     0     3.493721e+00     1.949049e+03
     0     3.493721e+00     1.949049e+03
     0     3.493721e+00     1.949049e+03
     0     3.493721e+00     1.949049e+03
 * time: 0.011278867721557617
 * time: 0.01034092903137207
 * time: 0.010904073715209961
 * time: 0.01090693473815918
     1     3.090287e+00     2.541144e+00
     1     3.090287e+00     2.541144e+00
 * time: 24.226807117462158
     1     3.090287e+00     2.541144e+00
 * time: 24.24724793434143
     1     3.090287e+00     2.541144e+00
 * time: 24.238826990127563
 * time: 24.223412036895752
     2     2.670622e+00     1.909469e-01
     2     2.670622e+00     1.909469e-01
 * time: 30.47301197052002
     2     2.670622e+00     1.909469e-01
 * time: 30.493449926376343
     2     2.670622e+00     1.909469e-01
 * time: 30.48503279685974
 * time: 30.46961498260498
     3     2.647692e+00     5.272250e-01
     3     2.647692e+00     5.272250e-01
 * time: 34.207842111587524
     3     2.647692e+00     5.272250e-01
 * time: 34.22828006744385
     3     2.647692e+00     5.272250e-01

As you can see, the optimization trace is a mess.

This specific example regards Optim, however, if there is a general solution that can be implemented for any module loaded, it would be nice!
Do you have a solution?

2

You can redirect_stdout for every process to it’s own file. For example:

using MPI

MPI.Init()

const comm = MPI.COMM_WORLD
const rank = MPI.Comm_rank(comm)

redirect_stdout(open("rank-$(rank).stdout", "a")) do
    print("Hello from rank $(rank)\n")
end

results in

$ mpiexecjl -n 2 julia main.jl

$ cat rank-0.stdout
Hello from rank 0

$ cat rank-1.stdout
Hello from rank 1

If you want to to it for the entire process (instead of locally as in the example above) just put

redirect_stdout(open("rank-$(rank).stdout", "a"))

or similar at the top of the script.

2 Likes
3

Thanks a lot!
This goes toward the direction I’m looking for and it could be already the solution.
Is it possible to hack this function to redirect the stdout of the rank 0 to the actual stdout while all other ranks to /dev/null or something similar?

4

@fredrikekre

Thanks a lot!
This goes toward the direction I’m looking for and it could be already the solution.
Is it possible to hack this function to redirect the stdout of the rank 0 to the actual stdout while all other ranks to /dev/null or something similar?

Yes, I found the solution

using MPI
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)

if rank != 0
	redirect_stdout(devnull)
end

println("Hello from rank $rank")

In this way all the print not from rank zero is redirected to /dev/null while the master can print on standard stdout. Thanks a lot!

5 
redirect_stdout(rank == 0 ? stdout : devnull) do
    print("Hello from rank $(rank)\n")
end

Edit: Didn’t see your second comment, looks like you already got it.

1 Like