# Codegen woes

**URL:** <https://discourse.julialang.org/t/codegen-woes/27098>\
**Category:** Performance\
**Created:** [August 2, 2019, 1:34am UTC](https://discourse.julialang.org/t/codegen-woes/27098 "2019-08-02T01:34:23Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [August 2, 2019, 1:34am UTC](https://discourse.julialang.org/t/codegen-woes/27098/1 "2019-08-02T01:34:23Z")

</div>

Hello all,

There’s an expressiveness problem I’ve been fighting with for quite a while now. I had thought (and hoped, really) I might just be missing something obvious. But from discussions at JuliaCon, it seems at least @dpsanders, @jpfairbanks, and @ChrisRackauckas have had some similar problems.

I’ll try to boil the problem down to a simple case. There’s a risk in doing this I might reduce the problem to one that has an easy but non-generalizable solution, or that I might not quite align with others’ use cases. Anyway, here goes…

Say your library has a function `makecode` that takes an `x::T` and produces an `Expr`. You’d like the user of your library to be able to pass their own `x` (or more likely, one they build using other tools you provide) and generate fast code.

There have been a few suggestions that might address this in at least some situations:

- @ChrisRackauckas had earlier suggested there may be a way to use nested macro calls to do this. But at this point I may not have been understanding the problem well enough to describe it well, so I don’t know if this is still a candidate.
- @mohamed82008 suggested expressing `T` in the type system, in order to use a generated function. This is an elegant solution, but type-level programming presents its own challenges. Even if this is the preferred approach, it would be great to be able to prototype DSLs more quickly than this approach allows.
- @tim.holy suggested (maybe jokingly?) that hacking the method table could help to get this working.
- The obvious goto (and my current approach) uses `invokelatest`. This is generally discouraged.

It seems there must be a way, however hacky, to allow this to be done easily and efficiently. My understanding of `invokelatest` had been that it creates a sort of boundary that’s expensive to cross but harmless otherwise.

But this seems not to be the case. Here’s a weird little example that attempts to abstract away some of my current workflow:

```julia
f(x) = quote
    function foo(a)
        a + $x
    end
end
​
function g(x)
    fx = f(x)
​
    quote
        $fx
​
        function h(start)
            s = start
            for j=1:10000
                s += foo(j)
            end
            return s
        end 
    end |> eval
​
    Base.invokelatest(h, x)
end

```

Performance is… not great:

```julia
julia> @btime g(2)
  7.831 ms (23946 allocations: 1.35 MiB)
50025002

```

Any ideas?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 1:44am UTC](https://discourse.julialang.org/t/codegen-woes/27098/2 "2019-08-02T01:44:57Z")

</div>

I posted a solution for a fast “invokelastest-like” function here:

[https://github.com/JuliaLang/julia/pull/32737](https://github.com/JuliaLang/julia/pull/32737)

I think there are real use cases for a function like this, but of course it could be abused.

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [August 2, 2019, 2:16am UTC](https://discourse.julialang.org/t/codegen-woes/27098/3 "2019-08-02T02:16:29Z")

</div>

Seems my approach has other problems, `invokefrozen(h, Int, x)` actually does worse. Or maybe it’s because I’m on v1.1.1?

```julia
julia> @btime g(2)
  8.559 ms (25957 allocations: 1.46 MiB)
50025002

```

I’ll need to do some profiling

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 2:29am UTC](https://discourse.julialang.org/t/codegen-woes/27098/4 "2019-08-02T02:29:32Z")

</div>

You’re measuring compilation time because every call to `g` is compiling. I have no idea how it would do w.r.t. compile time, it just has good runtime speed. You’d generate `h` once and then call that repeatedly in a loop, right? You want to measure your `h` calls, not the `h` compilation if that’s the case.

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [August 2, 2019, 2:30am UTC](https://discourse.julialang.org/t/codegen-woes/27098/5 "2019-08-02T02:30:53Z")

</div>

> [@ChrisRackauckas](#):
>
> You’re measuring compilation time

Haha so I am 🤦‍♂️

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 2:39am UTC](https://discourse.julialang.org/t/codegen-woes/27098/6 "2019-08-02T02:39:06Z")

</div>

I was testing how to take the symbolic derivative of a user’s function and build a function for that. The test I was running was:

```julia
using ModelingToolkit, BenchmarkTools
f(x) = x^4 - 3x^3 + 46*(x-x-x-x)
@variables x

function to_expr(O)
  if O isa ModelingToolkit.Constant
    return O.value
  elseif isa(O.op, Variable)
    isempty(O.args) && return O.op.name
    return Expr(:call, O.op,name, to_expr.(O.args)...)
  end
  return Expr(:call, Symbol(O.op), to_expr.(O.args)...)
end

function generate(O)
  if isa(O.op, Variable)
    isempty(O.args) && return O.op.name
    return Expr(:call, O.op,name, to_expr.(O.args)...)
  end
  return Expr(:call, Symbol(O.op), to_expr.(O.args)...)
end

function differentiate1(f)
  @variables x
  @derivatives D'~x
  op = f(x)
  op2 = expand_derivatives(D(op))
  ex2 = to_expr(op2)
  eval(:($(x.op.name) -> $ex2))
end

function _differentiate(f)
  @variables x
  @derivatives D'~x
  op = f(x)
  op2 = expand_derivatives(D(op))
  ex2 = to_expr(op2)
  _f = eval(quote
    $(x.op.name) -> begin
    $ex2
    end
  end)
  (rt,x) -> superdeadlyunsafe_invokelatest(_f,rt,x)
end

@inline @generated function superdeadlyunsafe_invokelatest(f, ::Type{rt}, args...) where rt
  tupargs = Expr(:tuple,args...)
  quote
    _f = $(Expr(:cfunction, Base.CFunction, :f, rt, :((Core.svec)($args...)), :(:ccall)))
    return ccall(_f.ptr,rt,$tupargs,$((:(getindex(args,$i)) for i in 1:length(args))...))
  end
end

function differentiate2(f)
  _f = _differentiate(f)
  x -> _f(typeof(x),x)
end

_df1 = differentiate1(f)
_df2 = differentiate2(f)
_df1(6)
_df2(6)
_df1(6.0)
_df2(6.0)

@btime $_df1($6) # 12
@btime $_df2($6) # 12
@btime $_df1($6.0) # 12
@btime $_df2($6.0) # 12

```

and other experiments here: [https://github.com/JuliaDiffEq/ModelingToolkit.jl/pull/155#issuecomment-515735993](https://github.com/JuliaDiffEq/ModelingToolkit.jl/pull/155#issuecomment-515735993) . This measures out to be 5ns overhead over the function itself, vs 60ns.

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [August 2, 2019, 2:40am UTC](https://discourse.julialang.org/t/codegen-woes/27098/7 "2019-08-02T02:40:24Z")

</div>

> [@ChrisRackauckas](#):
>
> This measures out to be 5ns overhead over the function itself, vs 60ns.

That’s really nice. I love the idea of passing the return type as an argument, that makes a lot of sense

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 9:32am UTC](https://discourse.julialang.org/t/codegen-woes/27098/8 "2019-08-02T09:32:19Z")

</div>

@ChrisRackauckas your solution still uses `eval` to define the function body. Any idea on how to eliminate the use of that?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 10:08am UTC](https://discourse.julialang.org/t/codegen-woes/27098/9 "2019-08-02T10:08:16Z")

</div>

No, it’s using Julia’s compiler at runtime so it needs it. Is there anything wrong with it other than aesthetics?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 10:26am UTC](https://discourse.julialang.org/t/codegen-woes/27098/10 "2019-08-02T10:26:24Z")

</div>

Inference in other functions?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 10:27am UTC](https://discourse.julialang.org/t/codegen-woes/27098/11 "2019-08-02T10:27:34Z")

</div>

But this infers fine?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 10:28am UTC](https://discourse.julialang.org/t/codegen-woes/27098/12 "2019-08-02T10:28:40Z")

</div>

Ok so it is almost a function barrier but with `eval`. Interesting…

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 10:30am UTC](https://discourse.julialang.org/t/codegen-woes/27098/13 "2019-08-02T10:30:45Z")

</div>

With the return type required to be given by the user, and then the compiler handles it just fine.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 10:34am UTC](https://discourse.julialang.org/t/codegen-woes/27098/14 "2019-08-02T10:34:45Z")

</div>

What about the scope of `eval`? Which module is it evaluating in? `Main`?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 10:41am UTC](https://discourse.julialang.org/t/codegen-woes/27098/15 "2019-08-02T10:41:10Z")

</div>

Depends on where you want it to eval. It’s the same eval.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 10:46am UTC](https://discourse.julialang.org/t/codegen-woes/27098/16 "2019-08-02T10:46:29Z")

</div>

Yes but I think in @cscherrer’s case, the body of the function `eval`ed can have arbitrary user code including code from modules not visible inside the module defining the function which calls `eval`. This inevitably means that we cannot `eval` in the defining module. So we can either always `eval` in `Main` which limits the use to REPL and the likes, not inside other modules, or `eval` in an input module but I don’t know if this would have any side effects.

To give a concrete example, let’s say `Soss.jl` has a function `transform_and_sample` that given a model definition, transforms the body, `eval`s the model, and samples using some MCMC alg. Then `DiffEqBayes` wants to define a model with a DiffEq solver in it and call `Soss.jl`. The model would then need to be `eval`ed in `DiffEqBayes` not `Soss` or `Main`.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 10:53am UTC](https://discourse.julialang.org/t/codegen-woes/27098/17 "2019-08-02T10:53:27Z")

</div>

Why not just use `Soss.eval` or `Main.eval`? I put the eval usage on the user side so they can make the right choice.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [August 2, 2019, 10:57am UTC](https://discourse.julialang.org/t/codegen-woes/27098/18 "2019-08-02T10:57:04Z")

</div>

I think we are using 2 definitions for “user”. The user that you mean is probably the user of the above function who is a package developer, e.g. Chad. The user that I mean is someone who wants to define a model and pass it to some function `Soss.sample` to do its magic. The second user shouldn’t have to explicitly `eval` the model IMO. But maybe that’s a small price to pay. So calling `eval` is on the end user.

And if that’s the case, then every non-exported name in the body of the model needs to have its namespace with it, e.g. `Soss.func`.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 2, 2019, 11:34am UTC](https://discourse.julialang.org/t/codegen-woes/27098/19 "2019-08-02T11:34:04Z")

</div>

I’m not sure of a better solution so I went with this one. Somewhere along the line someone needs to choose where the generated generics live, and to always get generic functions “working” the way the user wants it seems you need to let the user make the choice.

> [@mohamed82008](#):
>
> And if that’s the case, then every non-exported name in the body of the model needs to have its namespace with it, e.g. `Soss.func` .

Indeed that is the case, and you see that in the generators of ModelingToolkit:

> <https://github.com/SciML/ModelingToolkit.jl/blob/master/src/utils.jl#L62>

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 2, 2019, 11:46am UTC](https://discourse.julialang.org/t/codegen-woes/27098/20 "2019-08-02T11:46:36Z")

</div>

You can kinda use FunctionWrappers.jl for this as well

```julia
using FunctionWrappers: FunctionWrapper
using BenchmarkTools

function foo(expr)
    g = @eval function f(x, y)
        $expr
    end
    gwrap = FunctionWrapper{Int, Tuple{Int, Int}}(g)
    @btime $gwrap(1,2)
end

 foo(:(x + y))
 # 29.306 ns (0 allocations: 0 bytes)

```

[Next page](https://discourse.julialang.org/t/codegen-woes/27098.md?page=2)