# Compiling large Symbolics.jl expressions using build\_function

**URL:** <https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563>\
**Category:** Modelling & Simulations\
**Created:** [April 17, 2023, 6:13am UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563 "2023-04-17T06:13:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zornsllama](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zornsllama/32/24636_2.png) [@zornsllama](https://discourse.julialang.org/u/zornsllama)\
**Post date:** [April 17, 2023, 6:13am UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563/1 "2023-04-17T06:13:48Z")

</div>

Hi,

I’m using Symbolics.jl to compute derivatives for a nonlinear optimization problem, to be used as part of the Ipopt callback functions. I am ultimately hoping to match the performance of the C++ code generated in [this example](https://github.com/melizalab/dpse-example/), which uses SymPy to compute derivatives.

I noticed that when using `Symbolics.build_function` to turn a large symbolic expression into a Julia function, despite 40 min of waiting, the generated function never actually compiled. In contrast, when using the `Symbolics.CTarget()` option to generate a C function, the same expression immediately compiled and the generated function was quite fast and allocation-free.

I’m wondering why there is this discrepancy, and if there’s anything I can do to alleviate the issue in pure Julia. Worst case scenario, I can just make C functions and call them from Julia, but I am curious if the entire project can be done in pure Julia. In particular, is it possible to write equally efficient versions of the callback functions to those in the repository without leaving Julia? For reference, here is a [link](https://justpaste.it/c9g5e) to the generated C++ code, to give a sense of the type of callback function I am trying to implement.

Any help would be greatly appreciated! Thanks!

---

<div class="post-metadata">

**Author:** ![Vaibhavdixit02](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vaibhavdixit02/32/2916_2.png) [@Vaibhavdixit02](https://discourse.julialang.org/u/Vaibhavdixit02)\
**Post date:** [April 17, 2023, 7:23am UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563/2 "2023-04-17T07:23:28Z")

</div>

Do you need to do the derivative calculation and optimization yourself? If you are not constrained by that you could give Optimizaition.jl’s [`AutoModelingToolkit`](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#Optimization.AutoModelingToolkit) or the [symbolic interface](https://docs.sciml.ai/Optimization/stable/tutorials/symbolic/) if the model’s written up in MTK a try. To use Ipopt you’d use the OptimizationMOI wrapper [MathOptInterface.jl · Optimization.jl](https://docs.sciml.ai/Optimization/stable/optimization_packages/mathoptinterface/)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 17, 2023, 12:40pm UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563/3 "2023-04-17T12:40:29Z")

</div>

Symbolics.jl functions are not made to scale at this moment to tens of thousands of variables in the function building process. Something will be released pretty soon which improves this scaling though.

And are you testing on Julia v1.9 or v1.10? There have been some improvements in lowering large blocks (not all are merged though)

---

<div class="post-metadata">

**Author:** ![zornsllama](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zornsllama/32/24636_2.png) [@zornsllama](https://discourse.julialang.org/u/zornsllama)\
**Post date:** [April 17, 2023, 6:20pm UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563/4 "2023-04-17T18:20:58Z")

</div>

@Vaibhavdixit02: unfortunately I think I do need to perform the derivative calculations by hand. I’ve written up the nonlinear program as a JuMP model, and it does not scale well to the size of problem that I am ultimately interested in solving. I am also aware of [MathOptSymbolicAD](https://github.com/odow/MathOptSymbolicAD.jl), but was unable to get it working at a reasonable speed. There is also the concern that my model is large enough that I would like to perform the differentiation once, save the results somehow (e.g. compiled functions in a sysimage), and then reuse them as needed.

@ChrisRackauckas: for what it’s worth, the expression I’m directly trying to build a function out of has only around ~100 variables. More precisely, I’m trying to perform direct collocation of an ODE; each model constraint imposed by this only depends on a few timesteps, and the shape of the constraint is constant across timesteps. So I am trying to differentiate a single timestep constraint and use this result as a fast subroutine in building the entire sparse Jacobian/Hessian of Lagrangian. I am currently working in Julia 1.8.5.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 17, 2023, 11:14pm UTC](https://discourse.julialang.org/t/compiling-large-symbolics-jl-expressions-using-build-function/97563/5 "2023-04-17T23:14:03Z")

</div>

Have you tried Enzyme v0.11? That should do okay on this kind of thing. Indeed we have a lot of stuff coming for this pretty soon because a direct collocation into JuMP is rarely what you’d want to do, but the better stuff won’t be ready until summer so SparseDiffTools+Enzyme is the best solution for now.