Hi, I want to pre-compile functions which I use frequently from REPL. I use PackageCompiler#sd-notomls. It works well with compile_package("PyPlot"), but fails when list of precompiles is defined: compile_incremental("/home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl". Isn’t that strange, because if I am correct, compile_package uses same runtests.jl file to get precompiles?
julia> compile_incremental("/home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl")
ERROR: LoadError: LoadError: UndefVarError: version not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
[2] top-level scope at logging.jl:319
[3] include at ./boot.jl:326 [inlined]
[4] include_relative(::Module, ::String) at ./loading.jl:1038
[5] include at ./sysimg.jl:29 [inlined]
[6] include(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl:16
[7] top-level scope at none:0
[8] include at ./boot.jl:326 [inlined]
[9] include_relative(::Module, ::String) at ./loading.jl:1038
[10] include(::Module, ::String) at ./sysimg.jl:29
[11] exec_options(::Base.JLOptions) at ./client.jl:267
[12] _start() at ./client.jl:436
[13] jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1571 [inlined]
[14] true_main at /buildworker/worker/package_linux64/build/ui/repl.c:96
[15] main at /buildworker/worker/package_linux64/build/ui/repl.c:217
in expression starting at /home/kest/.julia/packages/PyPlot/cdCMF/test/runtests.jl:5
in expression starting at /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl:18
ERROR: failed process: Process(`/home/kest/Downloads/julia-1.1.1/bin/julia --optimize=3 --output-o=/home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/sys.a -g1 --track-allocation=none --startup-file=no --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --project=/home/kest/.julia/environments/v1.1 --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/home/kest/Downloads/julia-1.1.1/lib/julia/sys.so -g1 --compiled-modules=yes --optimize=2 /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:785 [inlined]
[3] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
[4] run at ./process.jl:724 [inlined]
[5] #run_julia#1 at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/compiler_flags.jl:225 [inlined]
[6] #run_julia at ./none:0 [inlined]
[7] #compile_incremental#57(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:137
[8] compile_incremental(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:134
[9] top-level scope at none:0
I’m not sure what exactly you’re doing… You’re talking about a list of precompiles, but seem to just use the runtests.jl? Did you try compile_incremental("PyPlot")?
Isn’t that strange, because if I am correct, compile_package uses same runtests.jl file to get precompiles?
It’s not really strange, since you don’t call compile_incremental with a file, but rather with a package just like compile_package
Sorry, I think I confused precompiles file with snoopfile, I didn’t find any example.
From compile_package(..) description:
<…> Compile a list of packages. Each package comes as a tuple of (package_name, precompile_file)
where the precompile file should contain all function calls, that should get compiled into the system image. Usually the runtests.jl file is a good candidate, since it should run all important functions of a package<…>
Is it possible to specify snoopfile in compile_incremental or compile_package args, for one (or more) packages? (in order to reduce compile time, by compiling only specified functions, not all from package’s runtests.jl)
MWE and short description of arguments would be great in PackageCompiler readme or functions descriptions.
There are a few manual steps that you need to do if you want to compile from your own snoop file, that isn’t currently refactored into it’s own API.
I actually just recently compiled from a custom snoopfile, here are the steps:
Thanks for reply, I am using yours example, just little modified to compile Plots and GR (for MWE), but error appears:
Generating precompiles..
Generating precompiles2..
Compile compile_incremental..
signal (11): Segmentation fault
in expression starting at no file:0
runtime_sym_gvs<LazyModule<emit_plt(jl_codectx_t&, llvm::FunctionType*, const llvm::AttributeList&, llvm::CallingConv::ID, char const*, char const*)::<lambda()> >&> at /buildworker/worker/package_linux64/build/src/ccall.cpp:98 [inlined]
emit_plt at /buildworker/worker/package_linux64/build/src/ccall.cpp:307
emit_a_ccall at /buildworker/worker/package_linux64/build/src/ccall.cpp:1978
emit_ccall at /buildworker/worker/package_linux64/build/src/ccall.cpp:1802
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:3967
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3662
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:3854
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1144
jl_compile_hint at /buildworker/worker/package_linux64/build/src/gf.c:1949
jl_compile_specializations at /buildworker/worker/package_linux64/build/src/precompile.c:378 [inlined]
jl_precompile at /buildworker/worker/package_linux64/build/src/precompile.c:387 [inlined]
jl_write_compiler_output at /buildworker/worker/package_linux64/build/src/precompile.c:34
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:222
main at /buildworker/worker/package_linux64/build/ui/repl.c:218
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /home/kest/Downloads/julia-1.1.1/bin/julia (unknown line)
Allocations: 98127619 (Pool: 98096916; Big: 30703); GC: 209
ERROR: LoadError: failed process: Process(`/home/kest/Downloads/julia-1.1.1/bin/julia --optimize=3 --output-o=/home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/sys.a -g1 --track-allocation=none --startup-file=no --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/home/kest/Downloads/julia-1.1.1/lib/julia/sys.so -g1 --compiled-modules=yes --optimize=3 /home/kest/.julia/packages/PackageCompiler/UQ5UO/sysimg/run_julia_code.jl`, ProcessSignaled(11)) [0]
Stacktrace:
[1] pipeline_error at ./process.jl:785 [inlined]
[2] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
[3] run at ./process.jl:724 [inlined]
[4] #run_julia#1 at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/compiler_flags.jl:225 [inlined]
[5] #run_julia at ./none:0 [inlined]
[6] #compile_incremental#57(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:137
[7] compile_incremental(::String) at /home/kest/.julia/packages/PackageCompiler/UQ5UO/src/incremental.jl:134
[8] top-level scope at none:0
in expression starting at /home/kest/Projects/gjJulia/blinkUI/snoopCompile.jl:28
snoop.jl:
using Plots, GR
gr()
Plots.plot(rand(10))
snoopCompile.jl:
##
println("Generating precompiles..")
using PackageCompiler
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles.jl", "w") do io
PackageCompiler.snoop(abspath("/home/kest/Projects/gjJulia/blinkUI/snoop.jl"), io)
end
## add the correct usings + imports to the precompile file
println("Generating precompiles2..")
using Pkg
packages = ["Plots", "GR"]
ctx = Pkg.Types.Context()
pkgs = PackageSpec.(packages)
packages = PackageCompiler.flat_deps(ctx, packages)
# remove blacklisted packages from full list of packages
imports = PackageCompiler.to_pkgid.(packages)
filter!(x-> !(x.name in ("WinRPM", "HTTPClient")), imports)
usings = join(["const $(x.name) = Base.require($(PackageCompiler.prepr(x)))" for x in imports], "\n")
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles2.jl", "w") do io
println(io, usings)
open("/home/kest/Projects/gjJulia/blinkUI/results/precompiles.jl") do io2
write(io, io2)
end
end
##
println("Compile compile_incremental..")
compile_incremental("/home/kest/Projects/gjJulia/blinkUI/results/precompiles2.jl")
println("Done.")
Now it worked, just used yours PackageCompiler’s snooping.jl functions to process snoopfile.jl in GR package folder.
Also snooping.jl tries to process direct_test_deps but I am using snoopfile.jl instead runtests.jl, it gives error, so I commented out this line as temporally workaround:
Why following snoopfile.jl file runs fine when launched in julia (using JIT compiler), but compile_incremental(:Makie) fails when using this file as snoopfile? (How briefly JIT ir AOT compilation sequences differs)
