Distributing a function that uses FFTW

General Usage
,
1

Hi! I’ve been having a lot of trouble using pmap with a function that uses FFTW with a planned FFT. Here is a minimum example, where I just try to perform a FFT on an array of matrices:

@everywhere using FFTW
using Distributed
addprocs(1)
pl = plan_fft(zeros(4, 4), flags=FFTW.MEASURE)
x = [zeros(4,4) for i in 1:2]
pmap(A -> pl * A, x)

I get a segmentation fault:

      From worker 2:	signal (11): Segmentation fault
      From worker 2:	in expression starting at none:0
      From worker 2:	fftw_execute_dft at...
 From worker 2:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 2:	#7 at ./In[5]:3
      From worker 2:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 2:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
      From worker 2:	do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
      From worker 2:	#106 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
      From worker 2:	run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
      From worker 2:	macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
      From worker 2:	#105 at ./task.jl:411
      From worker 2:	unknown function (ip: 0x7ff9e1f9796c)
      From worker 2:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 2:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
      From worker 2:	start_task at /buildworker/worker/package_linux64/build/src/task.c:833
      From worker 2:	Allocations: 8213770 (Pool: 8210931; Big: 2839); GC: 11

I’ve found a GitHub issue here that discusses this issue. However, I’m still mystified about how to solve it. I’ve tried plastering @everywhere’s everywhere and defining the providing function to pmap on every processor rather than making it anonymous. But I haven’t got things to work, much less figured out what would be the most appropriate way of doing this sort of thing in my larger scale program. I suppose I’m still a novice to how to write distributed code in Julia, in particular in how to pass information to workers (should I pass information through closures in pmap? or should I define things everywhere, and if so how without ruining my code? etc.) I would greatly appreciate your help!

2

The code seems to work if I do:

@everywhere using FFTW
using Distributed
addprocs(1)
@everywhere pl = plan_fft(zeros(4, 4), flags=FFTW.MEASURE)
x = [ones(4,4) for i in 1:2]
@everywhere f(A) = pl * A
pmap(A -> f(2*A), x)

Based on the GitHub issue, it seems like FFTW plans are not serializable, which causes the seg fault. This fixes the problem here, but what can I do in general? In my code, I do things such as create a struct which has an FFT plan as an attribute. Will I need to put an @everywhere on every line in the entire pipeline of code that contains my FFT plan? (e.g. in the example above, I needed to put an @everywhere on the function f. This seems really hard to do in my codebase, and also seems to go against Julia’s usual model of distributed computation. Please let me know if there’s a better way.