# PackageCompiler compile\_package works, but compile\_incremental fails on same runtests
**URL:** https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087
**Category:** General Usage
**Tags:** package
**Created:** [July 7, 2019, 9:30am UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087 "2019-07-07T09:30:16Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@KestutisMa](https://discourse.julialang.org/u/KestutisMa)
#### Post date: [July 7, 2019, 9:30am UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/1 "2019-07-07T09:30:16Z")
Hi, I want to pre-compile functions which I use frequently from REPL. I use PackageCompiler#sd-notomls. It works well with `compile_package("PyPlot")`, but fails when list of precompiles is defined: `compile_incremental("/home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl"`. Isn’t that strange, because if I am correct, compile\_package uses same runtests.jl file to get precompiles?
```julia
julia> compile_incremental("/home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl")
ERROR: LoadError: LoadError: UndefVarError: version not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
[2] top-level scope at logging.jl:319
[3] include at ./boot.jl:326 [inlined]
[4] include_relative(::Module, ::String) at ./loading.jl:1038
[5] include at ./sysimg.jl:29 [inlined]
[6] include(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl:16
[7] top-level scope at none:0
[8] include at ./boot.jl:326 [inlined]
[9] include_relative(::Module, ::String) at ./loading.jl:1038
[10] include(::Module, ::String) at ./sysimg.jl:29
[11] exec_options(::Base.JLOptions) at ./client.jl:267
[12] _start() at ./client.jl:436
[13] jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1571 [inlined]
[14] true_main at /buildworker/worker/package_linux64/build/ui/repl.c:96
[15] main at /buildworker/worker/package_linux64/build/ui/repl.c:217
in expression starting at /home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl:5
in expression starting at /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl:18
ERROR: failed process: Process(`/home/kest/Downloads/julia-1.1.1/bin/julia --optimize=3 --output-o=/home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/sys.a -g1 --track-allocation=none --startup-file=no --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --project=/home/kest/.julia/environments/v1.1 --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/home/kest/Downloads/julia-1.1.1/lib/julia/sys.so -g1 --compiled-modules=yes --optimize=2 /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:785 [inlined]
[3] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
[4] run at ./process.jl:724 [inlined]
[5] #run_julia#1 at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/compiler_flags.jl:225 [inlined]
[6] #run_julia at ./none:0 [inlined]
[7] #compile_incremental#57(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:137
[8] compile_incremental(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:134
[9] top-level scope at none:0
```
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [July 7, 2019, 12:58pm UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/2 "2019-07-07T12:58:16Z")
I’m not sure what exactly you’re doing… You’re talking about a list of precompiles, but seem to just use the runtests.jl? Did you try `compile_incremental("PyPlot")`?
> Isn’t that strange, because if I am correct, compile\_package uses same runtests.jl file to get precompiles?
It’s not really strange, since you don’t call `compile_incremental` with a file, but rather with a package just like `compile_package` 😉
---
### Author:  [@KestutisMa](https://discourse.julialang.org/u/KestutisMa)
#### Post date: [July 8, 2019, 1:51pm UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/3 "2019-07-08T13:51:12Z")
Sorry, I think I confused precompiles file with snoopfile, I didn’t find any example.
From `compile_package(..)` description:
> \<…\> Compile a list of packages. Each package comes as a tuple of `(package_name, precompile_file)`
> where the precompile file should contain all function calls, that should get compiled into the system image. Usually the `runtests.jl` file is a good candidate, since it should run all important functions of a package\<…\>
Is it possible to specify snoopfile in compile\_incremental or compile\_package args, for one (or more) packages? (in order to reduce compile time, by compiling only specified functions, not all from package’s runtests.jl)
MWE and short description of arguments would be great in PackageCompiler readme or functions descriptions. 😉
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [July 8, 2019, 1:54pm UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/4 "2019-07-08T13:54:18Z")
There are a few manual steps that you need to do if you want to compile from your own snoop file, that isn’t currently refactored into it’s own API.
I actually just recently compiled from a custom snoopfile, here are the steps:
> **[WGLMakie Image](https://nextjournal.com/sdanisch/wglmakie-image)**
---
### Author:  [@KestutisMa](https://discourse.julialang.org/u/KestutisMa)
#### Post date: [July 10, 2019, 3:19am UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/5 "2019-07-10T03:19:31Z")
Thanks for reply, I am using yours example, just little modified to compile Plots and GR (for MWE), but error appears:
```julia
Generating precompiles..
Generating precompiles2..
Compile compile_incremental..
signal (11): Segmentation fault
in expression starting at no file:0
runtime_sym_gvs >&> at /buildworker/worker/package_linux64/build/src/ccall.cpp:98 [inlined]
emit_plt at /buildworker/worker/package_linux64/build/src/ccall.cpp:307
emit_a_ccall at /buildworker/worker/package_linux64/build/src/ccall.cpp:1978
emit_ccall at /buildworker/worker/package_linux64/build/src/ccall.cpp:1802
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:3967
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3662
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:3854
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1144
jl_compile_hint at /buildworker/worker/package_linux64/build/src/gf.c:1949
jl_compile_specializations at /buildworker/worker/package_linux64/build/src/precompile.c:378 [inlined]
jl_precompile at /buildworker/worker/package_linux64/build/src/precompile.c:387 [inlined]
jl_write_compiler_output at /buildworker/worker/package_linux64/build/src/precompile.c:34
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:222
main at /buildworker/worker/package_linux64/build/ui/repl.c:218
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /home/kest/Downloads/julia-1.1.1/bin/julia (unknown line)
Allocations: 98127619 (Pool: 98096916; Big: 30703); GC: 209
ERROR: LoadError: failed process: Process(`/home/kest/Downloads/julia-1.1.1/bin/julia --optimize=3 --output-o=/home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/sys.a -g1 --track-allocation=none --startup-file=no --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/home/kest/Downloads/julia-1.1.1/lib/julia/sys.so -g1 --compiled-modules=yes --optimize=3 /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl`, ProcessSignaled(11)) [0]
Stacktrace:
[1] pipeline_error at ./process.jl:785 [inlined]
[2] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
[3] run at ./process.jl:724 [inlined]
[4] #run_julia#1 at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/compiler_flags.jl:225 [inlined]
[5] #run_julia at ./none:0 [inlined]
[6] #compile_incremental#57(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:137
[7] compile_incremental(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:134
[8] top-level scope at none:0
in expression starting at /home/kest/Projects/gjJulia/blinkUI/snoopCompile.jl:28
```
snoop.jl:
```julia
using Plots, GR
gr()
Plots.plot(rand(10))
```
snoopCompile.jl:
```julia
##
println("Generating precompiles..")
using PackageCompiler
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles.jl", "w") do io
PackageCompiler.snoop(abspath("/home/kest/Projects/gjJulia/blinkUI/snoop.jl"), io)
end
## add the correct usings + imports to the precompile file
println("Generating precompiles2..")
using Pkg
packages = ["Plots", "GR"]
ctx = Pkg.Types.Context()
pkgs = PackageSpec.(packages)
packages = PackageCompiler.flat_deps(ctx, packages)
# remove blacklisted packages from full list of packages
imports = PackageCompiler.to_pkgid.(packages)
filter!(x-> !(x.name in ("WinRPM", "HTTPClient")), imports)
usings = join(["const $(x.name) = Base.require($(PackageCompiler.prepr(x)))" for x in imports], "\n")
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles2.jl", "w") do io
println(io, usings)
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles.jl") do io2
write(io, io2)
end
end
##
println("Compile compile_incremental..")
compile_incremental("/home/kest/Projects/gjJulia/blinkUI/results/precompiles2.jl")
println("Done.")
```
Am I doing something wrong?
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [July 10, 2019, 8:36am UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/6 "2019-07-10T08:36:38Z")
Yeah, that’s the crux with doing this manually… Package compilation has lots of little gotchas.
GR is one of the packages that needs to be initialized:
[https://github.com/JuliaLang/PackageCompiler.jl/blob/sd-notomls/src/snooping.jl#L4](https://github.com/JuliaLang/PackageCompiler.jl/blob/sd-notomls/src/snooping.jl#L4)
So you’ll need to insert something like:
[https://github.com/JuliaLang/PackageCompiler.jl/blob/sd-notomls/src/snooping.jl#L164](https://github.com/JuliaLang/PackageCompiler.jl/blob/sd-notomls/src/snooping.jl#L164)
into your precompile file!
---
### Author:  [@KestutisMa](https://discourse.julialang.org/u/KestutisMa)
#### Post date: [July 10, 2019, 5:28pm UTC](https://discourse.julialang.org/t/packagecompiler-compile-package-works-but-compile-incremental-fails-on-same-runtests/26087/7 "2019-07-10T17:28:02Z")
Now it worked, just used yours PackageCompiler’s `snooping.jl` functions to process `snoopfile.jl` in GR package folder.
Also snooping.jl tries to process `direct_test_deps` but I am using snoopfile.jl instead runtests.jl, it gives error, so I commented out this line as temporally workaround:
>
Why following `snoopfile.jl` file runs fine when launched in julia (using JIT compiler), but `compile_incremental(:Makie)` fails when using this file as snoopfile? (How briefly JIT ir AOT compilation sequences differs)
snoopfile.jl:
```julia
using Makie
Makie.plot(rand(10))
```