# Best practices for profiling compilation

**URL:** <https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249>\
**Category:** Performance\
**Tags:** compilation\
**Created:** [October 12, 2020, 6:21pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249 "2020-10-12T18:21:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 12, 2020, 6:21pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/1 "2020-10-12T18:21:28Z")

</div>

In DataFrames.jl we see a strong trend that people want more and more powerful functions allowing for transformation of data frames (to match dplyr, data.table, pandas, …).

This leads to a situation when code base becomes really complex - which means long compilation. Additionally transformation functions most often take anonymous functions as arguments (so this is a “run once” case very often - most of the time the user will not call the transformation twice with the same argument types as every anonymous function has its own new type).

We experiment with various options how to use e.g. `@nospecialize` to reduce compilation latency, while ensuring that the code runs fast for large data.

However the thing I struggle with is a convenient way to check what and how gets compiled when a call to a function with a new signature happens. It would be great to hear back from the experts in the field what are the recommended approaches to it (currently I do it mostly based on my general understanding how things work and trial-and-error).

Thank you!

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 13, 2020, 1:40pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/2 "2020-10-13T13:40:14Z")

</div>

Didn’t see this before I posted [Understanding precompilation and its limitations (reducing latency)](https://discourse.julialang.org/t/understanding-precompilation-and-its-limitations-reducing-latency/48294), but that addresses some of what you’re asking about.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 13, 2020, 1:51pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/3 "2020-10-13T13:51:07Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 13, 2020, 2:31pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/4 "2020-10-13T14:31:22Z")

</div>

Also, just in general terms this is pretty much the main purpose of SnoopCompile. `@snoopi` measures inference, `@snoopc` codegen (native code generation), and `@snoopr` invalidations.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 13, 2020, 2:46pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/5 "2020-10-13T14:46:05Z")

</div>

One more thing: it looks like DataFrames/CategoricalArrays still invalidates an awful lot of MethodInstances. If you invalidate something that you also depend on, good luck getting precompilation to work.

Even if you don’t add any precompile directives, eliminating invalidations is also probably your easiest way to reduce latency. Anything you invalidated was already compiled, and if some of those MethodInstances are also useful to DataFrames then you’re just going to have to compile them again. If you can avoid that, you save both the inference time and the codegen time. (`precompile` directives currently only save you inference time; nice, but it’s even better if you can save both.)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 13, 2020, 3:40pm UTC](https://discourse.julialang.org/t/best-practices-for-profiling-compilation/48249/6 "2020-10-13T15:40:16Z")

</div>

The first step is to remove CategoricalArrays.jl as a dependency of DataFrames.jl and we are almost there on master already. Thank you for looking into this.