I tried to precompile Makie, because it seemed easy :
# add PackageCompiler
Pkg.add("PackageCompiler")
using PackageCompiler
# This is not well tested, so please be careful
# The safe option:
PackageCompiler.compile_package("Makie", force = false) # can take around ~20 minutes
But I ran into issues.
I tried to precompile in a project (not the default (v1.0) pkg>), where I installed Makie#master and PackageCompiler and ran the above commands, but it was missing both packages. It’s not a big problem, I may missed something about how PackageCompiler works.
Then I moved to the default project and added every needed package (for Makie). One of my issues is, that it takes like an hour till I run into an Error, not really the 20 minute mentioned in the docs. It’s running on my laptop with i7-3632QM, SSD and 8GB RAM, so lacking resources should not be an issue. Does it take that much time, or am I doing somethong wrong?
This is my main problem: while compiling, it is always missing some package, and as it takes an hour till I got an error, it is a really long time till I add every needed package. These are the packages that I had to install and wasn’t mentioned anywhere: ModernGL, MakieGallery, GLFW, ImageMagick, RData, Observables, RDatasets.
Julia and package versions:
julia> versioninfo()
Julia Version 1.0.2
Commit d789231e99 (2018-11-08 20:11 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)
(v1.0) pkg> st
Status `C:\Users\cstamas\.julia\environments\v1.0\Project.toml`
[537997a7] AbstractPlotting v0.9.4 #master (https://github.com/JuliaPlots/AbstractPlotting.jl.git)
[c52e3926] Atom v0.7.10
[6e4b80f9] BenchmarkTools v0.4.2
[336ed68f] CSV v0.4.3
[be33ccc6] CUDAnative v1.0.1
[5ae59095] Colors v0.9.5
[a6e380b2] ControlSystems v0.5.0
[150eb455] CoordinateTransformations v0.5.0
[3a865a2d] CuArrays v0.9.0
[1f15a43c] CxxWrap v0.8.1
[a93c6f00] DataFrames v0.17.0
[b4f34e82] Distances v0.7.4
[4f61f5a4] FFTViews v0.2.0
[7a1cc6ca] FFTW v0.2.4
[5789e2e9] FileIO v1.0.5
[587475ba] Flux v0.7.2
[f7f18e0c] GLFW v2.3.0
[e9467ef8] GLMakie v0.0.4 #master (https://github.com/JuliaPlots/GLMakie.jl.git)
[28b8d3ca] GR v0.37.0
[4d00f742] GeometryTypes v0.7.2
[7073ff75] IJulia v1.16.0
[6218d12a] ImageMagick v0.7.1
[70c4c096] Indicators v0.4.0
[c601a237] Interact v0.9.0
[e5e0dc1b] Juno v0.5.4
[b964fa9f] LaTeXStrings v1.0.3
[ee78f7c6] Makie v0.9.1 #master (https://github.com/JuliaPlots/Makie.jl.git)
[dbd62bd0] MakieGallery v0.0.2
[931e9471] MechanismGeometries v0.1.2
[283c5d60] MeshCat v0.5.0
[6ad125db] MeshCatMechanisms v0.2.1
[7269a6da] MeshIO v0.3.1
[66fc600b] ModernGL v1.0.0
[b8a86587] NearestNeighbors v0.4.3
[47be7bcc] ORCA v0.2.0
[9b87118b] PackageCompiler v0.6.1
[f0f68f2c] PlotlyJS v0.12.3
[91a5bcdd] Plots v0.23.0
[c46f51b8] ProfileView v0.4.0
[438e738f] PyCall v1.18.5
[d330b81b] PyPlot v2.7.0
[df47a6cb] RData v0.6.0
[a223df75] Reactive v0.8.3
[731186ca] RecursiveArrayTools v0.20.0
[366cf18f] RigidBodyDynamics v1.3.0
[6038ab10] Rotations v0.10.0
[90137ffa] StaticArrays v0.10.2
[2913bbd2] StatsBase v0.27.0
[0f1e0344] WebIO v0.4.2
To sum up my questions: 1) do PackageCompiler run only in the “main” project? 2) Is it normal that it takes so much time to precompile? 3) Which packages are needed for Makie to precompile? (this is my main question)
2 Likes
There’s a bug in the latest released version of PackageCompiler.jl with detecting test only dependencies, which I think is what you’re running into. I think that one is fixed on master, and if not, @sdanisch is probably on it
I’d also heavily recommend to use compile_incremental(:Makie)much faster and works with your existing sysimg instead of generating a new one from scratch.
Thank you for the answer!PackagCompiler#master and the results are the same. But I managed to add every dependency and the compilation ran succesfully. But it took several hours, which I don’t really understand (unless the compile time comment is outdated).force=true yet.)
You should try compile_incremental, which is quite a bit faster:
You need to start julia with julia J path/to/image.
x = PackageCompiler.sysimg_folder("sys.so")
y = unsafe_string(Base.JLOptions().image_file)
cp(x, y, force = true)
Don’t do the above without backing up your current system image
5 Likes
I narrowed down which pacakges are needed for the precompilation for easier use (at the time of this post):
add Makie#master
add AbstractPlotting#master
add GLMakie#master
add PackageCompiler#master
add ModernGL MakieGallery GLFW ImageMagick RData Observables RDatasets StatsMakie GeometryTypes Colors
Not sure about if PackageCompiler#master is needed, but it worked so it stays.
Also I got a snoop file error during the compilation:
julia> using PackageCompiler
julia> PackageCompiler.compile_incremental(:Makie, :AbstractPlotting, force = false)
Updating registry at `C:\Users\cstamas\.julia\registries\General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Makie\Project.toml`
[537997a7] + AbstractPlotting v0.9.5
[5789e2e9] + FileIO v1.0.5
[e9467ef8] + GLMakie v0.0.4
[6218d12a] + ImageMagick v0.7.1
[dca85d43] + QuartzImageIO v0.5.2
[44cfe95a] + Pkg
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Makie\Manifest.toml`
[621f4979] + AbstractFFTs v0.3.2
[537997a7] + AbstractPlotting v0.9.5
[13072b0f] + AxisAlgorithms v0.3.0
[39de3d68] + AxisArrays v0.3.0
[9e28174c] + BinDeps v0.8.10
[b99e7846] + BinaryProvider v0.5.3
[631607c0] + CMake v1.1.1
[d5fb7624] + CMakeWrapper v0.2.2
[a2cac450] + ColorBrewer v0.4.0
[3da002f7] + ColorTypes v0.7.5
[c3611d14] + ColorVectorSpace v0.6.2
[5ae59095] + Colors v0.9.5
[34da2185] + Compat v1.5.1
[8f4d0f93] + Conda v1.2.0
[d38c429a] + Contour v0.5.1
[150eb455] + CoordinateTransformations v0.5.0
[864edb3b] + DataStructures v0.15.0
[7a1cc6ca] + FFTW v0.2.4
[5789e2e9] + FileIO v1.0.5
[53c48c17] + FixedPointNumbers v0.5.3
[b38be410] + FreeType v2.1.1
[663a7486] + FreeTypeAbstraction v0.4.2
[f7f18e0c] + GLFW v2.3.0
[e9467ef8] + GLMakie v0.0.4
[4d00f742] + GeometryTypes v0.7.2
[a2bd30eb] + Graphics v0.4.0
[d9be37ee] + Homebrew v0.7.1
[bbac6d45] + IdentityRanges v0.3.0
[2803e5a7] + ImageAxes v0.5.0
[a09fc81d] + ImageCore v0.7.4
[6218d12a] + ImageMagick v0.7.1
[02fcd773] + ImageTransformations v0.7.2
[9b13fd28] + IndirectArrays v0.5.0
[a98d9a8b] + Interpolations v0.11.2
[8197267c] + IntervalSets v0.3.1
[c8e1da08] + IterTools v1.1.1
[682c06a0] + JSON v0.20.0
[1914dd2f] + MacroTools v0.4.4
[dbb5928d] + MappedArrays v0.2.1
[e1d29d7a] + Missings v0.4.0
[66fc600b] + ModernGL v1.0.0
[77ba4419] + NaNMath v0.3.2
[510215fc] + Observables v0.2.3
[6fe1bfb0] + OffsetArrays v0.10.0
[bac558e1] + OrderedCollections v1.0.2
[19eb6ba3] + Packing v0.3.0
[5432bcbf] + PaddedViews v0.4.2
[d96e819e] + Parameters v0.10.3
[995b91a9] + PlotUtils v0.5.5
[27ebfcd6] + Primes v0.4.0
[dca85d43] + QuartzImageIO v0.5.2
[b3c3ace0] + RangeArrays v0.3.1
[c84ed2f1] + Ratios v0.3.0
[189a3867] + Reexport v0.2.0
[6038ab10] + Rotations v0.11.1
[992d4aef] + Showoff v0.2.1
[73760f76] + SignedDistanceFields v0.4.0
[699a6c99] + SimpleTraits v0.8.0
[a2af1166] + SortingAlgorithms v0.3.1
[276daf66] + SpecialFunctions v0.7.2
[90137ffa] + StaticArrays v0.10.2
[2913bbd2] + StatsBase v0.27.0
[30578b45] + URIParser v0.4.0
[1cfade01] + UnicodeFun v0.4.0
[81def892] + VersionParsing v1.1.3
[efce3f68] + WoodburyMatrices v0.4.1
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8bb1440f] + DelimitedFiles
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Resolving package versions...
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Makie\Project.toml`
[9b87118b] + PackageCompiler v0.6.3
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Makie\Manifest.toml`
[c7e460c6] + ArgParse v0.6.1
[e1450e63] + BufferedStreams v1.0.0
[0862f596] + HTTPClient v0.2.1
[b27032c2] + LibCURL v0.4.1
[522f3ed2] + LibExpat v0.5.0
[2ec943e9] + Libz v1.0.0
[9b87118b] + PackageCompiler v0.6.3
[b718987f] + TextWrap v0.3.0
[c17dfb99] + WinRPM v0.4.2
[ Info: activating new environment at C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Makie\Project.toml.
Updating registry at `C:\Users\cstamas\.julia\registries\General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
[ Info: downloading reference images for version 0.0.9
[ Info: starting from index 1
[ Info: Caching fonts, this may take a while. Needed only on first run!
running tutorial_simple_scatter
running tutorial_markersize
running tutorial_simple_line
running tutorial_adding_to_a_scene
running tutorial_adjusting_scene_limits
running tutorial_basic_theming
running tutorial_heatmap
running tutorial_linesegments
running tutorial_barplot
running test_heatmap___image_overlap
running animation
running barplot_1
running quiver_1
running image_1
running scatter_colormap
running lots_of_heatmaps
running fem_polygon_2d
running fem_mesh_2d
running colored_triangle
running heatmap_interpolation
running colored_triangle_1
running subscenes
running polygons
running contour_function
running hbox_1
running customize_axes
running contour_1
running heatmap_1
running animated_scatter
running text_annotation
running text_rotation
running the_famous_iris_example
running chess_game
running parallel_prefix_sum
Scene (500px, 500px):
events:
window_area: GeometryTypes.HyperRectangle{2,Int64}([0, 0], [0, 0])
window_dpi: 100.0
window_open: false
mousebuttons: Set(AbstractPlotting.Mouse.Button[])
mouseposition: (0.0, 0.0)
mousedrag: notpressed
scroll: (0.0, 0.0)
keyboardbuttons: Set(AbstractPlotting.Keyboard.Button[])
unicode_input: Char[]
dropped_files: String[]
hasfocus: false
entered_window: false
plots:
*Axis3D{...}
*Mesh{...}
subscenes:running orthographic_camera
running volume_function
running textured_mesh
running load_mesh
running colored_mesh
running wireframe_of_a_mesh
running wireframe_of_sphere
running wireframe_of_a_surface
running surface_1
running surface_with_image
running line_function
running meshscatter_function
┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│ exception =
│ LoadError: MethodError: no method matching light_calc(::Observables.Observable{Bool})
│ Closest candidates are:
│ light_calc(!Matched::Bool) at C:\Users\cstamas\.julia\packages\GLMakie\rAuGK\src\GLVisualize\visualize\surface.jl:53
│ in expression starting at C:\Users\cstamas\.julia\packages\Makie\03coZ\test\runtests.jl:20
└ @ Main C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\sysimg\run_julia_code.jl:12
[ Info: used 7629 out of 7687 precompile statements
Updating registry at `C:\Users\cstamas\.julia\registries\General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\AbstractPlotting\Project.toml`
[a2cac450] + ColorBrewer v0.4.0
[3da002f7] + ColorTypes v0.7.5
[5ae59095] + Colors v0.9.5
[d38c429a] + Contour v0.5.1
[5789e2e9] + FileIO v1.0.5
[53c48c17] + FixedPointNumbers v0.5.3
[b38be410] + FreeType v2.1.1
[663a7486] + FreeTypeAbstraction v0.4.2
[4d00f742] + GeometryTypes v0.7.2
[6218d12a] + ImageMagick v0.7.1
[8197267c] + IntervalSets v0.3.1
[510215fc] + Observables v0.2.3
[19eb6ba3] + Packing v0.3.0
[995b91a9] + PlotUtils v0.5.5
[992d4aef] + Showoff v0.2.1
[73760f76] + SignedDistanceFields v0.4.0
[90137ffa] + StaticArrays v0.10.2
[1cfade01] + UnicodeFun v0.4.0
[37e2e46d] + LinearAlgebra
[d6f4376e] + Markdown
[de0858da] + Printf
[9a3f8284] + Random
[9e88b42a] + Serialization
[10745b16] + Statistics
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\AbstractPlotting\Manifest.toml`
[621f4979] + AbstractFFTs v0.3.2
[9e28174c] + BinDeps v0.8.10
[b99e7846] + BinaryProvider v0.5.3
[a2cac450] + ColorBrewer v0.4.0
[3da002f7] + ColorTypes v0.7.5
[c3611d14] + ColorVectorSpace v0.6.2
[5ae59095] + Colors v0.9.5
[34da2185] + Compat v1.5.1
[8f4d0f93] + Conda v1.2.0
[d38c429a] + Contour v0.5.1
[864edb3b] + DataStructures v0.15.0
[7a1cc6ca] + FFTW v0.2.4
[5789e2e9] + FileIO v1.0.5
[53c48c17] + FixedPointNumbers v0.5.3
[b38be410] + FreeType v2.1.1
[663a7486] + FreeTypeAbstraction v0.4.2
[4d00f742] + GeometryTypes v0.7.2
[a2bd30eb] + Graphics v0.4.0
[a09fc81d] + ImageCore v0.7.4
[6218d12a] + ImageMagick v0.7.1
[8197267c] + IntervalSets v0.3.1
[c8e1da08] + IterTools v1.1.1
[682c06a0] + JSON v0.20.0
[dbb5928d] + MappedArrays v0.2.1
[e1d29d7a] + Missings v0.4.0
[77ba4419] + NaNMath v0.3.2
[510215fc] + Observables v0.2.3
[6fe1bfb0] + OffsetArrays v0.10.0
[bac558e1] + OrderedCollections v1.0.2
[19eb6ba3] + Packing v0.3.0
[5432bcbf] + PaddedViews v0.4.2
[995b91a9] + PlotUtils v0.5.5
[189a3867] + Reexport v0.2.0
[992d4aef] + Showoff v0.2.1
[73760f76] + SignedDistanceFields v0.4.0
[a2af1166] + SortingAlgorithms v0.3.1
[276daf66] + SpecialFunctions v0.7.2
[90137ffa] + StaticArrays v0.10.2
[2913bbd2] + StatsBase v0.27.0
[30578b45] + URIParser v0.4.0
[1cfade01] + UnicodeFun v0.4.0
[81def892] + VersionParsing v1.1.3
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8bb1440f] + DelimitedFiles
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Resolving package versions...
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\AbstractPlotting\Project.toml`
[9b87118b] + PackageCompiler v0.6.3
[44cfe95a] + Pkg
Updating `C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\AbstractPlotting\Manifest.toml`
[c7e460c6] + ArgParse v0.6.1
[e1450e63] + BufferedStreams v1.0.0
[0862f596] + HTTPClient v0.2.1
[b27032c2] + LibCURL v0.4.1
[522f3ed2] + LibExpat v0.5.0
[2ec943e9] + Libz v1.0.0
[9b87118b] + PackageCompiler v0.6.3
[b718987f] + TextWrap v0.3.0
[c17dfb99] + WinRPM v0.4.2
[ Info: activating new environment at C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\AbstractPlotting\Project.toml.
Updating registry at `C:\Users\cstamas\.julia\registries\General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
[ Info: used 270 out of 270 precompile statements
┌ Info: activating new environment at C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\packages\Project.toml.
└ @ Pkg.API C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Pkg\src\API.jl:519
Updating registry at `C:\Users\cstamas\.julia\registries\General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: QuartzImageIO.jl can only be used on Apple macOS. Suggested usage is
│ @static if Sys.isapple()
│ using QuartzImageIO
│ # QuartzImageIO specific code goes here
│ end
└ @ QuartzImageIO C:\Users\cstamas\.julia\packages\QuartzImageIO\9DhKg\src\QuartzImageIO.jl:723
WARNING: using Contour.Contour in module anonymous conflicts with an existing identifier.
Build shared library "C:\\Users\\cstamas\\.julia\\packages\\PackageCompiler\\4vYdO\\sysimg\\sys.dll":
`'C:\Users\cstamas\.julia\packages\WinRPM\Y9QdZ\deps\usr\x86_64-w64-mingw32\sys-root\mingw\bin\gcc.exe' --sysroot 'C:\Users\cstamas\.julia\packages\WinRPM\Y9QdZ\deps\usr\x86_64-w64-mingw32\sys-root' -shared '-DJULIAC_PROGRAM_LIBNAME="C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\sysimg\sys.dll"' -o 'C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\sysimg\sys.dll' -Wl,--whole-archive 'C:\Users\cstamas\.julia\packages\PackageCompiler\4vYdO\sysimg\sys.a' -Wl,--no-whole-archive -std=gnu99 '-IC:\Users\cstamas\AppData\Local\Julia-1.1.0\include\julia' -DJULIA_ENABLE_THREADING=1 '-LC:\Users\cstamas\AppData\Local\Julia-1.1.0\bin' -Wl,--stack,8388608 -ljulia -lopenlibm -m64 -O3 -Wl,--export-all-symbols`
("C:\\Users\\cstamas\\.julia\\packages\\PackageCompiler\\4vYdO\\sysimg\\sys.dll", "C:\\Users\\cstamas\\AppData\\Local\\Julia-1.1.0\\lib\\julia\\sys.dll")
Is it a Makie or PackageCompiler error?
julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)
(v1.1) pkg> st
Status `C:\Users\cstamas\.julia\environments\v1.1\Project.toml`
[537997a7] AbstractPlotting v0.9.5 #master (https://github.com/JuliaPlots/AbstractPlotting.jl.git)
[5ae59095] Colors v0.9.5
[f7f18e0c] GLFW v2.3.0
[e9467ef8] GLMakie v0.0.4 #master (https://github.com/JuliaPlots/GLMakie.jl.git)
[4d00f742] GeometryTypes v0.7.2
[6218d12a] ImageMagick v0.7.1
[ee78f7c6] Makie v0.9.1 #master (https://github.com/JuliaPlots/Makie.jl.git)
[dbd62bd0] MakieGallery v0.0.4
[66fc600b] ModernGL v1.0.0
[510215fc] Observables v0.2.3
[9b87118b] PackageCompiler v0.6.3+ #master (https://github.com/JuliaLang/PackageCompiler.jl.git)
[df47a6cb] RData v0.6.0
[ce6b1742] RDatasets v0.6.1
[65254759] StatsMakie v0.0.2
1 Like
juliohm
September 29, 2021, 5:20pm
6
Could you please share up-to-date instructions to precompile WGLMakie.jl? We are writing Pluto notebooks using WGLMakie.jl visualizations and would like to assess the speedup gained by a PackageCompiler.jl generated sysimage.
cc: @sdanisch
1 Like
Did you try create_sysimage(["Makie", "WGLMakie"]; sysimage_path="sys.so")?
1 Like
