Patterns for precompiling workloads that could error

General Usage
,
1

I’ve been thinking about precompilation and code that could error. In this situation, I would like to run a workload that might throw an error during precompilation. The workload is just for optimization, an error here should not be fatal. You should still be able to load the package. This encourages me to compile more since the risk of making my package unusable due to errors is decreased.

An example workload might be one that write to disk. This workload could fail if the disk were full.

Objectives:

  1. Allow for successful precompilation even if my work load errors
  2. Succintly report that something went wrong
  3. Log the errors and stack trace for retrieval later
  4. Warn on package import that an error occurred during precompilation
  5. Create a mechanism for precompilation.

Here is the pattern I’m settling on to accomplish this.

module PC_G
    using PrecompileTools
    using Logging

    const precompile_log = Ref{String}()
    const precompile_exception = Ref{Union{Exception,Nothing}}(nothing)

    @setup_workload let
        # Setup Logging
        iob = IOBuffer()
        ioc = IOContext(iob, :color => true)
        logger = ConsoleLogger(ioc)

        @compile_workload try
            # Try to run some workloads, but allow failure
            throw(error("This is a non-fatal error during precompilaton"))
        catch err
            # Report an issue during precompilation
            @warn "An error occurred during precompilation of PC_G. Run `PC_G.print_precompile_log()` for details."
            with_logger(logger) do
                # Log the full stacktrace for later
                @error "An error occurred during precompilation of PC_G" exception=(err,catch_backtrace())
            end
            precompile_exception[] = err
        end
        precompile_log[] = String(take!(iob))
    end

    # Convenience function to print precompilation log
    function print_precompile_log(io = stdout)
        println(io, precompile_log[])
        @info "To recompile PC_G, run `PC_G.recompile()`"
    end

    # Force precompilation
    function recompile()
        Base.compilecache(Base.identify_package("PC_G"))
    end
end

Here is a demonstration of how this works in practice.

julia> using Pkg

julia> pkg"precompile" # A warning is generated
Precompiling project...
  1 dependency successfully precompiled in 3 seconds. 2 already precompiled.
  1 dependency had warnings during precompilation:
┌ PC_G [53d9e621-99c1-490b-9901-ab24cb4123b7]
│  ┌ Warning: An error occurred during precompilation of PC_G. Run `PC_G.print_precompile_log()` for details.
│  └ @ PC_G ~/.julia/dev/PrecompileTools/test/PC_G/src/PC_G.jl:19
└  

julia> using PC_G # The warning is also repeating on initial package import, with more details
┌ Warning: An error occurred during precompilation of PC_G
│   exception =
│    This is a non-fatal error during precompilaton
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] macro expansion
│        @ ~/.julia/dev/PrecompileTools/test/PC_G/src/PC_G.jl:16 [inlined]
│      [3] macro expansion
│        @ ~/.julia/packages/PrecompileTools/EqjW2/src/workloads.jl:74 [inlined]
│      [4] macro expansion
│        @ ~/.julia/dev/PrecompileTools/test/PC_G/src/PC_G.jl:14 [inlined]
│      [5] top-level scope
│        @ ~/.julia/packages/PrecompileTools/EqjW2/src/workloads.jl:136
│      [6] include
│        @ ./Base.jl:457 [inlined]
│      [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
│        @ Base ./loading.jl:2010
│      [8] top-level scope
│        @ stdin:2
│      [9] eval
│        @ ./boot.jl:370 [inlined]
│     [10] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│        @ Base ./loading.jl:1864
│     [11] include_string
│        @ ./loading.jl:1874 [inlined]
│     [12] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:305
│     [13] _start()
│        @ Base ./client.jl:522
└ @ PC_G ~/.julia/dev/PrecompileTools/test/PC_G/src/PC_G.jl:22

[ Info: To recompile PC_G, run `PC_G.recompile()`

julia> PC_G.recompile() # Precompilation can be redone, perhaps with non-code related conditions resolved
[ Info: Precompiling PC_G [53d9e621-99c1-490b-9901-ab24cb4123b7]
┌ Warning: An error occurred during precompilation of PC_G. Run `PC_G.print_precompile_log()` for details.
└ @ PC_G ~/.julia/dev/PrecompileTools/test/PC_G/src/PC_G.jl:19
("/home/mkitti/.julia/compiled/v1.9/PC_G/VoPgP_bL9aV.ji", "/home/mkitti/.julia/compiled/v1.9/PC_G/VoPgP_bL9aV.so")

Do you use a particular pattern for catching non-fatal errors in precompilation workloads?

1 Like