Function fails when used in precompile workload

General Usage
,
1

I have the following function:

function str2type(name)
    typename = Symbol(name)
    t = getfield(Main, typename)
    instance = t()
end

I need it to get an instance of a type of a marker struct at runtime based on strings in the setup.yaml file.

Normally this works fine, but it fails when I call it in the @setup_workload section of my package.

Example:

@setup_workload begin
    @compile_workload begin
        a = Velocity_Constant()
        b = str2type("Velocity_Constant")
    end
end

The assignment to a works fine, the assignment to b fails with the error:

julia> using FLORIDyn
Precompiling FLORIDyn...
Info Given FLORIDyn was explicitly requested, output will be shown live 
ERROR: LoadError: UndefVarError: `Velocity_Constant` not defined in `Main`
Stacktrace:
 [1] str2type(name::String)
   @ FLORIDyn ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:65
 [2] macro expansion
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:249 [inlined]
 [3] macro expansion
   @ ~/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:78 [inlined]
 [4] macro expansion
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:243 [inlined]
 [5] macro expansion
   @ ~/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:140 [inlined]
 [6] top-level scope
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:240
 [7] include
   @ ./Base.jl:562 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2881
 [9] top-level scope
   @ stdin:6
in expression starting at /home/ufechner/repos/FLORIDyn.jl/src/FLORIDyn.jl:4
in expression starting at stdin:6
  ✗ FLORIDyn
  0 dependencies successfully precompiled in 4 seconds. 119 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

FLORIDyn 

Failed to precompile FLORIDyn [fc770985-7669-4833-bc54-a38444eb6027] to "/home/ufechner/.julia/compiled/v1.11/FLORIDyn/jl_h52Xz5".
ERROR: LoadError: UndefVarError: `Velocity_Constant` not defined in `Main`
Stacktrace:
 [1] str2type(name::String)
   @ FLORIDyn ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:65
 [2] macro expansion
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:249 [inlined]
 [3] macro expansion
   @ ~/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:78 [inlined]
 [4] macro expansion
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:243 [inlined]
 [5] macro expansion
   @ ~/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:140 [inlined]
 [6] top-level scope
   @ ~/repos/FLORIDyn.jl/src/FLORIDyn.jl:240
 [7] include
   @ ./Base.jl:562 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2881
 [9] top-level scope
   @ stdin:6
in expression starting at /home/ufechner/repos/FLORIDyn.jl/src/FLORIDyn.jl:4
in expression starting at stdin:

What could be the reason?

(Full source code: FLORIDyn.jl/src at main · ufechner7/FLORIDyn.jl · GitHub)

2

Velocity_Constant seems to be a name in FLORIDyn, I guess it’s not present in Main during precompilation?

3

Shell I use a different module name than Main? I mean, the struct is defined, otherwise the line where a is assigned would not work.

4

Precompilation will spawn a new Julia process, so anything you define in the Main module can not be accessed when precompiling. Even neglecting precompilation, depending on the state of the user’s Main module like this seems like an unusual API choice though. For instance, an approach like this will be inherently type unstable. Why not let the user pass a type explicitly and parametrize on that?

5

I dispatch on marker structs, like:

struct Velocity_Constant <: VelModel end

The functions then look like this:

function getWindSpeedT(::Velocity_Constant, WindVel, iT, _)

I have many of these functions, they have the same name, but a different type as first parameter. Not my design choice, a consequence of translating Matlab code that was switching the search path for the functions at runtime.

My Settings constructor looks like this:

function Settings(wind::Wind, sim::Sim, con::Con)
    vel_mode = str2type("Velocity_" * wind.input_vel)
    dir_mode = str2type("Direction_" * wind.input_dir)
    turb_mode = str2type("TI_" * wind.input_ti)
    shear_mode = str2type("Shear_" * wind.input_shear)
    cor_dir_mode = str2type("Direction_" * wind.correction.dir)
    cor_vel_mode = str2type("Velocity_" * wind.correction.vel)
    cor_turb_mode = str2type("TI_" * wind.correction.ti)
    iterate_mode = str2type(sim.dyn.op_iteration)
    control_mode = str2type("Yaw_" * con.yaw)
    Settings(vel_mode, dir_mode, turb_mode, shear_mode, cor_dir_mode, cor_vel_mode, cor_turb_mode, 
             iterate_mode, control_mode)
end

So I really need the str2type function. The structs passed to the Settings constructor are created from a yaml settings file.

My only question is, how to write the str2type function such that I can use it during precompilation.

6

I changed the function to:

function str2type(name)
    typename = Symbol(name)
    t = getfield(FLORIDyn, typename)
    instance = t()
end

And now all works fine!

Thank you!

7

I still don’t understand why you need to create separate types if all that changes are the values in a yaml file. If you want your package to work with precompilation you will have to change your approach, as I said, as-is your design is not compatible with precompilation. I didn’t realize the types were actually defined in your package and not by the user. In that case your approach will work, but the issues around type-stability still stand

8

Yeah I agree with the change to the parent module of the fixed set of types. Using Main seems to weirdly restrict things to REPL work, which goes out the window if the user likes to change contextual modules often or happens to assign different things to the same names.

Not that it’d further help this resolved problem, but the PrecompileTools documentation says:

There are cases where you might want to precompile code but cannot safely execute that code: for example, you may need to connect to a database, or perhaps this is a plotting package but you may be currently on a headless server lacking a display, etc. In that case, your best option is to fall back on Julia’s own precompile function.

though it also explains that you’d need to do more work to discover and write more precompile calls for any useful runtime-dispatched callees.