I’m currently working on creating a package. I’ve got everything working but the time first using it in a project takes quite long, about 17 seconds. Packages that I’m using which are way larger take much less time to load. I know this topic has been discussed to death from the perspective of using someone else’s packages, but I’m wondering what I can do myself to speed up the loading of my package. Are there any best practices that I need to know of, or documents that I can read about this topic?
Step 1 is profiling. In Julia 1.8, you can run @time_imports using InteractiveIsing which will tell you how much of the time comes from packages you are loading vs your code.
1 Like
You have a LOT of big dependencies. Consider if you need them all.
Instead of DataFrames.jl/CSV.jl, just use the Tables.jl interface. Instead of Plots.jl, add recipes with RecipesBase.jl.
3 Likes
Sorry for the late reply, I’ve been on vacation in the meantime.
Thanks, though, I didn’t know about this. Here is the result:
@time_imports using InteractiveIsing
0.2 ms Requires
100.6 ms FileIO 5.05% compilation time (17% recompilation)
1.5 ms StaticArraysCore
393.2 ms StaticArrays
0.1 ms Reexport
40.9 ms FixedPointNumbers
49.2 ms ColorTypes 6.33% compilation time
131.3 ms Colors
4.0 ms IrrationalConstants
0.2 ms Compat
55.1 ms ChainRulesCore
2.2 ms DocStringExtensions 60.00% compilation time
0.4 ms ChangesOfVariables
0.5 ms InverseFunctions
0.5 ms LogExpFunctions
0.7 ms OpenLibm_jll
14.0 ms Preferences
0.3 ms JLLWrappers
5.9 ms CompilerSupportLibraries_jll
155.8 ms OpenSpecFun_jll 99.53% compilation time (100% recompilation)
11.2 ms SpecialFunctions
0.2 ms TensorCore
114.3 ms ColorVectorSpace 2.61% compilation time
0.3 ms Adapt
47.4 ms OffsetArrays
1.8 ms PaddedViews
1.8 ms MappedArrays
1.6 ms StackViews
0.6 ms MosaicViews
0.2 ms NaNMath
2.0 ms Graphics
7.3 ms AbstractFFTs
488.5 ms ImageCore
48.2 ms ImageBase
0.9 ms DataAPI
5.3 ms OrderedCollections
61.5 ms DataStructures
0.3 ms SortingAlgorithms
8.1 ms Missings
0.3 ms StatsAPI
16.9 ms StatsBase
2.3 ms IndirectArrays
5.1 ms CoordinateTransformations
1.3 ms Calculus
61.6 ms DualNumbers
10.4 ms Quaternions
198.9 ms Rotations
5.8 ms WoodburyMatrices
45.8 ms Ratios 74.05% compilation time
0.2 ms AxisAlgorithms
33.0 ms Interpolations 13.21% compilation time
3.3 ms ImageTransformations
35.5 ms MacroTools
1.2 ms SimpleTraits
2.9 ms RangeArrays
20.4 ms IntervalSets
22.6 ms IterTools
18.1 ms AxisArrays
3.5 ms ImageAxes
15.8 ms ImageMetadata
0.9 ms FFTW_jll
285.5 ms FFTW 4.44% compilation time (100% recompilation)
0.1 ms CustomUnitRanges
6.2 ms FFTViews
1.8 ms ComputationalResources
4.8 ms TiledIteration
5.9 ms CatIndices
304.4 ms ImageFiltering 1.13% compilation time
20.6 ms ImageMorphology 53.64% compilation time
3.7 ms Distances
1.9 ms ImageDistances
0.1 ms UnPack
0.4 ms Parameters
5.7 ms ImageContrastAdjustment
4.2 ms LazyModules 67.47% compilation time
1.0 ms ImageQualityIndexes
1.9 ms IntegralArrays
3.4 ms RegionTrees
2.7 ms NearestNeighbors
3.7 ms Clustering
3.1 ms ArnoldiMethod
0.4 ms Inflate
32.0 ms Graphs
1.5 ms TranscodingStreams
283.7 ms JLD2
3.5 ms MetaGraphs
32.5 ms SimpleWeightedGraphs
1.6 ms ImageSegmentation
1.5 ms ImageShow
5.3 ms Images
12.0 ms ColorSchemes
0.7 ms Zlib_jll
1.3 ms OpenSSL_jll
1.0 ms HDF5_jll
130.5 ms HDF5 68.58% compilation time
0.6 ms Zstd_jll
0.4 ms Lz4_jll
0.4 ms Blosc_jll
2.7 ms Blosc 76.65% compilation time
72.1 ms H5Zblosc 88.23% compilation time
163.1 ms JLD 59.95% compilation time (98% recompilation)
21.0 ms PDMats
29.9 ms Rmath_jll 97.87% compilation time (100% recompilation)
40.4 ms Rmath 84.19% compilation time
0.7 ms HypergeometricFunctions
4.9 ms StatsFuns
2.5 ms QuadGK
159.9 ms FillArrays
1.1 ms DensityInterface
198.4 ms Distributions
12.1 ms Observables
21.1 ms PooledArrays
3.6 ms InvertedIndices
0.1 ms IteratorInterfaceExtensions
0.1 ms TableTraits
0.5 ms Formatting
0.1 ms DataValueInterfaces
17.1 ms Tables
214.6 ms Crayons
233.0 ms PrettyTables
2183.6 ms DataFrames
127.6 ms Parsers 2.91% compilation time
167.4 ms SentinelArrays 22.74% compilation time
14.3 ms InlineStrings
24.1 ms WeakRefStrings
1.7 ms CodecZlib
12.4 ms FilePathsBase 17.14% compilation time
1654.0 ms CSV 78.55% compilation time (88% recompilation)
208.9 ms libcxxwrap_julia_jll 95.42% compilation time (100% recompilation)
766.0 ms CxxWrap 10.44% compilation time
0.9 ms LaTeXStrings
30.9 ms StructArrays
2.8 ms Extents
0.9 ms GeoInterface
315.5 ms EarCut_jll 99.60% compilation time (100% recompilation)
703.5 ms GeometryBasics
3.0 ms Unzip
0.5 ms Bzip2_jll
0.4 ms FreeType2_jll
0.3 ms FriBidi_jll
0.4 ms Libiconv_jll
0.2 ms Libffi_jll
0.7 ms XML2_jll
1.1 ms Gettext_jll
0.4 ms PCRE_jll
3.9 ms Glib_jll
0.5 ms Pixman_jll
0.3 ms libpng_jll
0.1 ms Libuuid_jll
0.4 ms Expat_jll
2.4 ms Fontconfig_jll 64.77% compilation time
0.3 ms LZO_jll
1.3 ms Cairo_jll
0.4 ms Graphite2_jll
0.9 ms HarfBuzz_jll
0.5 ms libass_jll
0.4 ms libfdk_aac_jll
0.5 ms LAME_jll
0.4 ms Ogg_jll
0.8 ms libvorbis_jll
0.7 ms libaom_jll
0.5 ms x264_jll
0.7 ms x265_jll
0.3 ms Opus_jll
5.1 ms FFMPEG_jll
0.2 ms FFMPEG
117.9 ms RecipesBase
435.6 ms PlotUtils
23.5 ms PlotThemes
0.9 ms UnicodeFun
0.1 ms Showoff
20.8 ms JSON
7.6 ms Measures
4.3 ms RecipesPipeline
0.2 ms Scratch
0.3 ms RelocatableFolders
414.8 ms GR 38.43% compilation time (100% recompilation)
3506.3 ms Plots 29.31% compilation time (66% recompilation)
0.1 ms Xorg_libXext_jll
0.1 ms Xorg_libxcb_jll
0.1 ms Xorg_xcb_util_wm_jll
0.1 ms Xorg_xcb_util_image_jll
0.1 ms Xorg_xcb_util_keysyms_jll
0.1 ms Xorg_xcb_util_renderutil_jll
0.1 ms Xorg_libXrender_jll
0.1 ms xkbcommon_jll
0.1 ms Libglvnd_jll
11.3 ms Qt6Base_jll
1.5 ms Qt6ShaderTools_jll
10.5 ms Qt6Declarative_jll
0.9 ms Qt6Svg_jll
14.6 ms jlqml_jll
229.8 ms QML 6.07% compilation time (11% recompilation)
273.8 ms InteractiveIsing 12.23% compilation time
It seems that, as Raf presumably already saw, indeed Dataframes and Plots are the worst offenders, though CSV and GR also take quite a long time. Are there any other ways of cutting down the loading time of these packages, other than just cutting them out? Is there no way, for example, to include compiled versions of them in my package? Something like including a sys image?
Also, two questions: Why does it seem like some packages don’t have to be compiled (or at least, no compilation time percentage is shown) and what is recompilation?