Pmap error with closure

General Usage
,
1

I’ve been getting an error having to do with pmap on closures that I don’t understand. Anyone know what is going on?

addprocs(2)

function gen_f(a::T) where T
    f = x -> T
    return f
end
f = gen_f(1f0) 

f(1f0)   # --> Float32
pmap(f, [1f0, 2f0])   # --> Error deserializing a remote exception from worker 2
2

Not really sure what’s happening here.

But it might be something to do with f having a type parameter:

Julia-0.6.2> typeof(f)
##14#15{Float32}

This does work for me:

function gen_g(a)
    f = x -> typeof(a)
    return f
end
g = gen_g(1f0) 

pmap(g, [1f0, 2f0])
3

Thanks for checking up on it. For my actual use case using typeof(a) isn’t very natural but it might be workable. Seems like a bug though so I guess I’ll post a report on github.

4

… now I recall the problem with using typeof(a), I loose type stability:

julia> function gen_f1(a)
           T = typeof(a)
           f = x -> T(x)
           return f
       end
gen_f1 (generic function with 1 method)

julia> function gen_f2(a::T) where T
           f = x -> T(x)
           return f
       end
gen_f2 (generic function with 1 method)

julia> f1 = gen_f1(1f0)
(::#5) (generic function with 1 method)

julia> f2 = gen_f2(1f0)
(::#7) (generic function with 1 method)

julia> @code_warntype f1(1f0)
Variables:
  #self#::##5#6{DataType}
  x::Float32

Body:
  begin
      return ((Core.getfield)(#self#::##5#6{DataType}, :T)::DataType)(x::Float32)::Any
  end::Any

julia> @code_warntype f2(1f0)
Variables:
  #self# <optimized out>
  x::Float32

Body:
  begin
      return x::Float32
  end::Float32
5

Could you use a generated function for gen_f?

6

I’m guessing your suggesting something like defining f directly as a generated function? Or are you thinking of using @generated for gen_f?

For my use case I’m defining multiple functions within one closure (where all the functions are sharing some local closure variables that are precomputed) which are all passed to the workers. At least I’m not aware of how to use @generated for that. Anyway, thanks for the suggestion.

7

Glad to see the issue being fixed.
That’s an amazing response!

Cross-referencing issue and PR for posterity:
https://github.com/JuliaLang/julia/issues/26979
https://github.com/JuliaLang/julia/pull/26986

8

Indeed! Although I’m somewhat embarrassed that I didn’t find that one liner fix. Thanks for posting the cross reference.

(Fingers crossed it gets into 0.6.3 backports.)