Precompilation techniques for speeding up the first run (Heroku deployments and PackageCompiler)

essenciary · September 30, 2019, 2:26pm

I’m using a Dockerfile to deploy a Genie app on a host and among other commands, I deploy the app with:

RUN julia -e "using Pkg; pkg\"activate . \"; pkg\"instantiate\"; pkg\"precompile\"; "

However, upon starting the app, it still takes about 30s for the first run (it’s a very small free Heroku server). What does that pkg"precompile" command do, because it doesn’t seem to help.

Thanks!

kristoffer.carlsson · September 30, 2019, 2:28pm

It runs the same thing that happens when the package is imported for the first time.

essenciary · September 30, 2019, 2:35pm

Thanks That just moved the problem somewhere else: “what happens when the package is imported for the first time”? Is there anything in the docs about it?

ninjaaron · September 30, 2019, 2:36pm

Possibly relevant: What's actually inside precompiled Julia files?

essenciary · September 30, 2019, 2:42pm

Thanks, that’s useful. I’ll try with SnoopCompile.

I’ve been running a lot of tests with PackageCompiler but I can’t automate the compilation. The main issue seems to be that is uses the tests files which use packages which are not dependencies of the main package (ie my app uses Plots and does not have VisualRegressionTests in its Manifest file – but PackageCompiler crashes because running the tests uses VisualRegressionTests).

My issue is similar to yours - I’m deploying web apps and I’m trying to reduce the first request time.

ninjaaron · September 30, 2019, 2:46pm

You could send a dummy request when the server starts. Not sure if that’s good enough for what you’re trying to do.

essenciary · September 30, 2019, 2:51pm

Yes, this is another option I’m considering. Though it’s a bit more complicated. Heroku powers down the server on the free tier when the app is idle - and automatically restarts it on first request. It takes about 3s on the first request just to restart the server - but the real problem is with Julia when it then takes some 30s to re-compile the app…

So I think I need to keep the app alive by not allowing it to go idle - which means making an automated request every minute or so, ideally from within the app itself (so I don’t need yet another external script to run forever).

ninjaaron · October 2, 2019, 6:55pm

So, I’m reading your post again after two days (I did read it the first time, of course), but I’m wondering if Heroku might not be the problem here? Might you be better served by a flat, always-up VPS?

I’m not the expert, but I know there is a docker-based Heroku clone called dokku which you can self-host. I can say that I’m very happy with DigitalOcean as my VPS for personal projects, but YMMV.

essenciary · October 3, 2019, 8:47am

I can’t objectively say that Heroku is the problem. Objectively, the problem is the long JIT-compilation time which can not be controlled/eliminated. The more complex the app and the larger packages it has, the longer it takes to (re)start it up.

I’m happy with DO but that’s not what I’m after. My objective is to demonstrate a happy path workflow - I’m working on documentation and a video showing how to bootstrap a Genie app from 0 to deployment on Heroku in a few minutes. The major advantage is that Heroku is user-friendly and has a free tier. In my Rails learning days, being able to build and publish an app with ease, was a magical experience. And Heroku was part of it.

I have plans to add support (integrations) for more major hosting providers, including DO. But the fact that Heroku provides a free tier where Julians could just whip up their projects and host them there during dev, with only 2 Genie commands, is really nice.

PS - I managed to address the issue of keeping it alive by adding functionality within Genie to optionally ping URLs (which can also be used for triggering the JIT-compilation so the first users don’t get a 30s delay on their requests).

However, the issue now is that the initial startup goes over 60s. And if the app doesn’t bind to the port in 60s it gets killed by Heroku. I was trying to build a nice and relevant demo but it seems that the only thing that would allow the app to run on the free tier is to remove Plots.jl and refactor the app without using Plots.jl.

We really need a way to speed up deployments/first-runs of Julia apps.

===

PS2 - @ninjaaron Dokku looks cool for their Azure, DO, and DH integrations. So maybe I can use it for Genie to provide deployments via Dokku. Thanks for sharing.

kristoffer.carlsson · October 3, 2019, 9:57am

In what way doesn’t PackageCompiler + a script that runs a representative workload not work?

Julia itself does this to “deploy” a julia with a fast REPL + Pkg etc. We literally run a little mini julia session that gets compiled and saved:

github.com

JuliaLang/julia/blob/888d32ab9ed99981c1894e43aca46aec3dc8ce4d/contrib/generate_precompile.jl#L16-L31


      
          precompile_script = """
          2+2
          print("")
          @time 1+1
          ; pwd
          ? reinterpret
          using Ra\t$CTRL_C
          \\alpha\t$CTRL_C
          \e[200~paste here ;)\e[201~"$CTRL_C
          $UP_ARROW$DOWN_ARROW$CTRL_C
          123\b\b\b$CTRL_C
          \b\b$CTRL_C
          f(x) = x03
          f(1,2)
          [][1]
          cd("complet_path\t\t$CTRL_C

essenciary · October 3, 2019, 10:29am

@kristoffer.carlsson Thanks

All the attempts I’ve made at building a custom sys image with PackageCompiler failed. Literally none worked. I used the PackageCompiler API and passed the names of the packages I wanted added to the sys image, with both full and incremental pre-compilation (as shown in the various examples).

The process always failed due to unresolved dependencies. For example, attempting to precompile Plots.jl always crashes due to missing VisualRegressionTests. I have to explicitly add a line to the Dockerfile to add VisualRegressionTests - and then it eventually crashes in another missing dependency, and so on. I’m sure it can be solved by manually adding the missing packages one by one, but my objective was to automate it and this approach won’t work.

===

Edit:
I guess the issue is that VisualRegressionTests is added as extras and not as deps:
https://github.com/JuliaPlots/Plots.jl/blob/2c3bffaf766a6ca9eaf0a3fd446e89269d0e9503/Project.toml

kristoffer.carlsson · October 3, 2019, 11:03am

Alright, so the answer to the question

We really need a way to speed up deployments/first-runs of Julia apps.

is not something big that needs to happen. It is just that PackageCompiler needs to work slightly better.

essenciary · October 3, 2019, 12:07pm

I expect that you’re in a much better position to judge if PackageCompiler is the best way to achieve faster deployments/first-runs of Julia apps. My understanding after diving into the topic of AOT/pre compilation for a few days was that there is no reliable way of doing it (as also confirmed by my experiments).

Ideally, I would love to see AOT compilation as an option in the compiler - do the development using JIT and publish using AOT.

I can’t really comment on the effectiveness of PackageCompiler as it didn’t work for me. So are you saying that, as far as you know, PackageCompiler is the best way to achieve faster deployments/first-runs of Julia apps?

kristoffer.carlsson · October 3, 2019, 12:52pm

This way PackageCompiler does things ends up being exactly how Julia itself becomes fast to start, so yes. As an example, let’s take Pkg:

With Pkg in sysimg:

julia> @time Pkg.status()
...
  0.034113 seconds (59.39 k allocations: 3.378 MiB, 6.75% gc time)

Using non-compiled Pkg:

julia> @time Pkg.status()
...
  1.415450 seconds (4.29 M allocations: 203.818 MiB, 5.40% gc time)

There is nothing special about Pkg. It is a normal julia package that has its own mini “training” script:

github.com

JuliaLang/Pkg.jl/blob/a10291a94d2e956dad980ccdc4dba17b1cf274e5/src/Pkg.jl#L517-L534


      
          const CTRL_C = '\x03'
          const precompile_script = """
              import Pkg
              tmp = mktempdir()
              cd(tmp)
              empty!(DEPOT_PATH)
              pushfirst!(DEPOT_PATH, tmp)
              # Prevent cloning registry
              mkdir("registries")
              touch("registries/blocker") # prevents default registry from cloning
              touch("Project.toml")
              ] activate .
              $CTRL_C
              Pkg.add("Test") # adding an stdlib doesn't require internet access
              ] add Te\t\t$CTRL_C
              ] st
              $CTRL_C
              rm(tmp; recursive=true)"""

which is run when the sysimg is created and the compiled code is saved.

Alexander-Barth · October 3, 2019, 1:14pm

I have similar issues with the Julia compile latency, but in all my tests with PackageCompiler.jl I ended up hitting a problem which I could not resolve. For example:

https://github.com/JuliaLang/PackageCompiler.jl/issues/184#issuecomment-464906938

essenciary · October 3, 2019, 1:27pm

Got it, thanks. That’s good to know - in this case, I will keep focusing on PackageCompiler.

essenciary · October 3, 2019, 7:20pm

Dropping this here in case other people will run into the Heroku 60s port binding timeout issue.

In the end my solution was a workaround: I implemented in Genie an option for early binding. This does two things:

very early in the application’s life cycle it binds a dummy Sockets.TCPServer onto the Heroku port:

const EARLYBINDING = Sockets.listen(parse(IPAddr, ENV["HOST"]), parse(Int, ENV["PORT"]))

This pleases the Heroku gods which no longer crush my defenseless app.

Later on, once the application is loaded, I start the fully configured app server (using HTTP.jl). Very important, I pass the EARLYBINDING TCPServer into HTTP.serve as the optional server argument:

HTTP.serve(..., server = EARLYBINDING)

Otherwise HTTP.serve will crash as the port will be already used by our decoy.

So I’m happy to report that now Heroku deployments on the puny free tier servers work very reliably and as expected from compiled Julia, performance is excellent. However, in some instances, the Heroku logs show that it can take up to 3 minutes to precompile the app

rikh · May 6, 2021, 1:25pm

For people stumbling into this thread: you can also increase the boot timeout to as much as 180 seconds at https://tools.heroku.support/limits/boot_timeout.

Also, to speed up the first HTTP.jl response, you could indeed do the PackageCompiler way or trigger compilation via an @async task. Via the latter, the service time went from more than 3 seconds to 2 milliseconds on the first GET request with the following code:

function trigger_compilation(port)
    sleep(1)
    router = M.router()
    routes = router.routes
    routes = [first(route) for route in routes]
    get_routes = filter(route -> route.method == "GET", routes)

    function hit_route(route)
        url = "http://$(Sockets.localhost):$(port)$(route.path)"
        HTTP.get(url)
    end

    hit_route.(get_routes)
end

function serve(host::String, port::Integer)
    println("Starting server at $(url(port))")
    socket = server_socket(host, port)
    @async trigger_compilation(port)
    HTTP.serve(router(), host, port; server = socket)
end

mbauman · May 6, 2021, 3:19pm

Oh man, I wish I would have known these tricks! This thread also predates my work here — which hacked into Optamatica’s buildpack to use PackageCompiler more thoroughly and attempts to bring the built package size down a bit:

I’ve not upstreamed the changes yet because I’m not totally happy about how it depends upon a precompile file generated by --trace-compile.