The real version is using PencilArrays.jl which are MPI arrays designed to work with FFTW and I see similar behavior. I was just going for an easy test problem for illustration. Your point about the multiple integrators I think probably is the key to the problem, although I can’t quite get my head around how to make it work in an MPI environment.
I can set he pencil arrays up for the problem, but then when I declare the integrator I face a dilemma: The kind of dumb way I did it results in every worker having their own integrator and fails whenever one of them fails an error check, but if only one worker creates the integrator then the code fails immediately.
if mpi_rank == 0
integrator = ...
step!(integrator)
end
fails because the other workers never see the call to the function so the boss node waits forever at the start of the rhs function.
if mpi_rank == 0
integrator = ...
end
step!(integrator)
will fail because all of the other workers see the integrator as an undefined variable. Hopefully I’m missing something obvious and someone can point it out. If not, I’ll keep playing with it and see where I get.