Hi!
My package (AugmentedGaussianProcesses.jl) always take forever to precompile when I make modifications. I would like to see where the bottlenecks are to eventually optimize or remove problematic stuff.
Is there anyway I could do this? I tried ProfileView, and Profile but they tell me they don’t collect any samples.
Thanks!
2 Likes
The issue is that precompilation happens in a separate Julia process (one that gets launched with special options --output-ji and --output-incremental). What if you navigate to the package src/ directory and then do @profile include("AugmentedGaussianProcesses.jl")?
You also might temporarily comment out the module AugmentedGaussianProcesses and corresponding end lines and then use time_showline from https://github.com/timholy/DebuggingUtilities.jl. That might give you a bit more insight of the kind you were hoping for.
Would be nice to be able to get a printout that shows how much the “self time” and total time of loading a package (+ deps) is.
So if you do using MegaMLFramework it would show something like:
name self total
MegaMLFramework 2 s 20 s
Flux 6 s 18 s
Zygote 12 s 12 s
Would be nice to be able to get a printout that shows how much the “self time” and total time of loading a package (+ deps) is.
You can get partway there with
julia> tid = Tuple{Float64,Base.PkgId}[]
0-element Array{Tuple{Float64,Base.PkgId},1}
julia> rectime(id) = push!(tid, (time(), id))
rectime (generic function with 1 method)
julia> push!(Base.package_callbacks, rectime)
1-element Array{Any,1}:
rectime
julia> using Images
[ Info: Precompiling Images [916415d5-f1e6-5110-898d-aaa5f9f070e0]
julia> tid
54-element Array{Tuple{Float64,Base.PkgId},1}:
(1.579204597981122e9, DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a])
(1.579204598355999e9, OpenSpecFun_jll [efe28fd5-8261-553b-a9e1-b2916fc3738e])
(1.579204598393891e9, SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b])
(1.579204598403032e9, NaNMath [77ba4419-2d1f-58cd-9bb1-8ffee604a2e3])
(1.579204598431032e9, OffsetArrays [6fe1bfb0-de20-5000-8ca7-80f57d26f881])
(1.579204598433714e9, PaddedViews [5432bcbf-9aad-5242-b902-cca2824c8663])
(1.579204598439408e9, MappedArrays [dbb5928d-eab1-5f90-85c2-b9b0edb7c900])
(1.579204598480289e9, Requires [ae029012-a4dd-5104-9daa-d747884805df])
(1.579204598575368e9, FixedPointNumbers [53c48c17-4a7d-5ca2-90c5-79b7896eea93])
(1.57920459910741e9, ColorTypes [3da002f7-5984-5a60-b8a6-cbb66c0b333f])
(1.579204628329869e9, Compat [34da2185-b29b-5c13-b0c7-acf172513d20])
(1.57920462834716e9, Reexport [189a3867-3050-52da-a836-e630ba90ab69])
(1.579204628387631e9, Colors [5ae59095-9a9b-59fe-a467-6f913c188581])
(1.579204628410532e9, Graphics [a2bd30eb-e257-5431-a919-1863eab51364])
(1.579204628565449e9, ImageCore [a09fc81d-aa75-5fe9-8630-4744c3626534])
(1.579204628585339e9, ImageMorphology [787d08f9-d448-5407-9aad-5290dd7ab264])
(1.579204628636617e9, OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d])
(1.579204628824953e9, DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8])
(1.579204628878409e9, Distances [b4f34e82-e78d-54a5-968a-f98e89d6e8f7])
(1.579204628933431e9, IntervalSets [8197267c-284f-5f27-9208-e0e47529a953])
(1.579204629006153e9, IterTools [c8e1da08-722c-5040-9ed9-7db0dc04731e])
(1.579204629034534e9, RangeArrays [b3c3ace0-ae52-54e7-9d0b-2c1406fd6b9d])
(1.579204629097144e9, AxisArrays [39de3d68-74b9-583c-8d2d-e117c070f3a9])
(1.579204629135714e9, WoodburyMatrices [efce3f68-66dc-5838-9240-27a6d6f5f9b6])
(1.579204629264411e9, MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09])
(1.579204629290151e9, SimpleTraits [699a6c99-e7fa-54fc-8d76-47d257e15c1d])
(1.5792046296115e9, StaticArrays [90137ffa-7385-5640-81b9-e52037218182])
(1.579204629626384e9, ImageAxes [2803e5a7-5153-5ecf-9a86-9b4c37f5f5ac])
(1.579204629656018e9, Missings [e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28])
(1.579204629658575e9, SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c])
(1.579204629724387e9, StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91])
(1.579204630044383e9, Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc])
(1.579204630071168e9, AxisAlgorithms [13072b0f-2c55-5437-9ae7-d433b7a33950])
(1.579204630071702e9, CustomUnitRanges [dc8bdbbb-1ca9-579f-8c36-e416f6a65cce])
(1.579204630109859e9, AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c])
(1.57920463105436e9, FFTW_jll [f5851436-0d7a-5f13-b9de-f02708fd171a])
(1.579204631167306e9, FFTW [7a1cc6ca-52ef-59f5-83cd-3a7055c09341])
(1.579204631853694e9, FFTViews [4f61f5a4-77b1-5117-aa51-3ab5ef4ef0cd])
(1.579204631869348e9, TiledIteration [06e1c1a7-607b-532d-9fad-de7d9aa2abac])
(1.579204631887049e9, ComputationalResources [ed09eef8-17a6-5b46-8889-db040fac31e3])
(1.579204631900806e9, Ratios [c84ed2f1-dad5-54f0-aa8e-dbefe2724439])
(1.579204631997281e9, Interpolations [a98d9a8b-a2ab-59e6-89dd-64a1c18fca59])
(1.579204632006697e9, IdentityRanges [bbac6d45-d8f3-5730-bfe4-7a449cd117ca])
(1.579204632034646e9, ColorVectorSpace [c3611d14-8923-5661-9e6a-0046d554d3a4])
(1.579204632040712e9, IndirectArrays [9b13fd28-a010-5f03-acff-a1bbcff69959])
(1.57920463227854e9, FileIO [5789e2e9-d7fb-5bc7-8068-2c6fae9b9549])
(1.579204632348105e9, ImageShow [4e3cecfd-b093-5904-9786-8bbb286a6a31])
(1.579204632406012e9, CoordinateTransformations [150eb455-5306-5404-9cee-2592286d6298])
(1.579204632444702e9, ImageTransformations [02fcd773-0e25-5acc-982a-7f6622650795])
(1.579204632492719e9, ImageDistances [51556ac3-7006-55f5-8cb3-34580c88182d])
(1.579204632534033e9, CatIndices [aafaddc9-749c-510e-ac4f-586e18779b91])
(1.579204632587353e9, ImageMetadata [bc367c6b-8a6b-528e-b4bd-a4b897500b49])
(1.579204632764635e9, ImageFiltering [6a3955dd-da59-5b1f-98d4-e7296123deb5])
(1.579204632808183e9, Images [916415d5-f1e6-5110-898d-aaa5f9f070e0])
Also, a bit off-topic, but @theogf just checking that you know about Revise.jl as an alternative means to circumvent part of this problem during your development workflow.
1 Like
@tim.holy, thanks a lot for the DebuggingUtilities.jl solution! It definitely did the trick!
I am aware of Revise.jl and I cannot praise enough this package. My precompilation problem comes from using different environments (with DrWatson.jl) for my experiments and that my package needs to be recompiled every time.
Well I found out that having Flux as a dependency was not the smartest idea. Even just using Flux.Optimise takes a very long time, I thought that it would just load the submodule, but it still needs to load the whole Flux precompilation.