# Package compilation woes

**URL:** <https://discourse.julialang.org/t/package-compilation-woes/5658>\
**Category:** General Usage\
**Tags:** package\
**Created:** [August 31, 2017, 2:30pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658 "2017-08-31T14:30:23Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [August 31, 2017, 2:30pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/1 "2017-08-31T14:30:23Z")

</div>

Here’s the situation:

- I have the HTTP.jl package, which includes a decent amount of fairly modular code
- Without any tricks, package compilation takes 7 seconds, using takes 0.7 s, and making a first request takes 5 whole seconds
- Trying SnoopCompile.jl didn’t actually seem to make any difference (maybe I’m using it wrong? Branch is [here](https://github.com/JuliaWeb/HTTP.jl/tree/jq/snoop))
- I also tried putting a block just after my HTTP definition like:

```julia
try
    HTTP.get("www.google.com")
end

```

but that didn’t seem to make any difference either.

- The option I’m considering going w/ is putting the above try-block in the module ` __init__ ()`; that balloons the `using` time to 10s, which is **_painful_** to have to pay every single time you do `using HTTP`, but in a production setting, I think it’s the only option vs. having the first request/response take 5 whole seconds to compile/process

Anybody have any other ideas? Feeling a little stuck here.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 31, 2017, 4:01pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/2 "2017-08-31T16:01:00Z")

</div>

I have no good idea how to solve this, but I think sticking it into `__init()__` would be really problematic: now any package that depends on HTTP.jl will have those 10s added when someone loads that third package via `using Foo`, even if the user of `Foo` actually never intends to use functionality in Foo.jl that depends on HTTP.jl.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [August 31, 2017, 4:12pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/3 "2017-08-31T16:12:51Z")

</div>

If that helps make the case and light the fire for figuring out a real solution to conditional package dependencies then I’m in! But seriously, we can’t keep dancing around this issue forever, adding weird workarounds and hacks, if it takes making a bunch of packages really painful, then it can only help prioritize the work to figure out a solution.

Even so, it’s simply unacceptable in any kind of production setting to have the first request/response take 5 seconds to process; it’s much better to require any compilation happen at package load time. Granted, these “production apps” could be the ones to put their own “warm up” code in their own ` __init__ ()`, but that’s easy to forget and a hassle. There’s just not a great solution to these issues right now as far as I can tell.

---

<div class="post-metadata">

**Author:** ![malmaud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/malmaud/32/29_2.png) [@malmaud](https://discourse.julialang.org/u/malmaud)\
**Post date:** [August 31, 2017, 5:12pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/4 "2017-08-31T17:12:19Z")

</div>

Can we do any kind of profiling of where that 5 seconds in the initial get is actually being spent? It seems weird since there isn’t _that_ much Julia code being compiled for `get`.

---

<div class="post-metadata">

**Author:** ![Michael\_Eastwood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/michael_eastwood/32/669_2.png) [@Michael\_Eastwood](https://discourse.julialang.org/u/Michael_Eastwood)\
**Post date:** [August 31, 2017, 6:23pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/5 "2017-08-31T18:23:40Z")

</div>

You have a typo in your SnoopCompile branch. You’re including `precompile_Base.jl` 4 times over. Fixing that might help, but also from the SnoopCompile README:

> Currently, precompilation does not cache functions from other modules; as a consequence, your speedup in execution time might be smaller than you’d like. In such cases, one strategy is to generate a script for your base/userimg.jl file and build the packages (with precompiles) into julia itself.

So the precompile statements for modules other than HTTP aren’t really winning you anything.

---

<div class="post-metadata">

**Author:** ![malmaud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/malmaud/32/29_2.png) [@malmaud](https://discourse.julialang.org/u/malmaud)\
**Post date:** [September 1, 2017, 12:58am UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/6 "2017-09-01T00:58:45Z")

</div>

I’ve experimented with SnoopCompile for my own packages and found the gain is modest at best, so it doesn’t surprise me that it isn’t helping here much.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [September 1, 2017, 11:54am UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/7 "2017-09-01T11:54:22Z")

</div>

I hope that this issue gets some attention. I have code where type inference lasts longer than a minute and don’t have an idea how to reduce it. Can’t share the code unfortunately.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 1, 2017, 1:51pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/8 "2017-09-01T13:51:20Z")

</div>

Oh but there kind of is…[https://github.com/JuliaWeb/HTTP.jl/blob/0bd57463bb95bb5fb45eb02a529c464772ee1ff2/src/parser.jl#L159](https://github.com/JuliaWeb/HTTP.jl/blob/0bd57463bb95bb5fb45eb02a529c464772ee1ff2/src/parser.jl#L159). It’s a behemoth 1000+ line function, ported from C.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 1, 2017, 5:13pm UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/9 "2017-09-01T17:13:01Z")

</div>

Update: spent a decent amount of time this morning digging in to some solutions and made some progress. Turns out the `precompile` statements generated from SnoopCompile weren’t even valid, but seemed to assume some old `call` convention; ensuring each `precompile` statement [was valid](https://github.com/JuliaWeb/HTTP.jl/commit/989b5bd938e1621efc42ef88f739c6a86c8d54a8) seems to have helped decently. I now see initial package compilation times between 10-20s, w/ subsequent package `using HTTP` down to 2.5-3s (just a hair more than w/o the precompile statements), and the initial `HTTP.get` request is down to 3s (from 6s).

I’ve also opened an issue [here](https://github.com/JuliaLang/julia/issues/23548) to gather some feedback/answers to additional things I came across in my journey to understanding precompile. Hopefully the answers/feedback there will continue to improve the situation, but for now, I at least think the situation is much more reasonable than it was.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [September 13, 2017, 12:00am UTC](https://discourse.julialang.org/t/package-compilation-woes/5658/10 "2017-09-13T00:00:00Z")

</div>

Hi @quinnj, how is this all going? Do you have anything that’s ready to be tested?