How to serialize a Julia function and define that function in another process?

[dfdx]

@jinliangwei

I see your point. I was actually intentionally looking for an exact match, a.k.a. log(b::Number, x::Number) in Base.Math at math.jl:186 in your example and would report an error if an exact match can’t be found.

Then you can extract all methods and filter them by type with something like on Julia 0.6:

function exact_signature(m::Method, f::Function, types::Tuple{Vararg{DataType}})
    return m.sig == Tuple{typeof(f), types...}
end

ms = collect(methods(log, (Number, Number)))
exact_signature(ms[1], log, (Number, Number))  # ==> false
exact_signature(ms[2], log, (Number, Number))  # ==> false
exact_signature(ms[3], log, (Number, Number))  # ==> true

I understood that I can get the AST of that function but I was actually asking what would be the best way to serialize that AST. I don’t think serialize works for this case as it can’t serialize Ptr across processes.

As far as I know, Julia doesn’t use pointers to represent any part of its AST. Main building blocks of ASTs are Expr, Symbol, data (e.g. numbers and arrays) and a few special structures like QuoteNode, all of which are perfectly serializable. The way you deserialize it, however, is important: if your serialized function refers to anything in a current process that you don’t pass along to another process, evaluating this function will of course fail. When I was working on Spark.jl (which was much earlier than Sugar.jl appeared on the radar), extracting dependencies was exactly the most complicated part. Eventually, I ended up with an explicit macro @attach that copied passed in expression to all workers, so that users don’t get confused about what is and what isn’t available on other processes.