Passing simulation parameters into app

I’m trying to use PackageCompiler to make my code into an app to share. The compiler is working, but it’s forcing me to modify how I deal with input parameters. The code takes in a number of inputs (including functions) and performs a simulation. Previously the inputs were defined in a namedTuple, which is passed into the solver as an argument.

With the compiled app, I am trying to pass in the name of a file that contains the namedTuple as an argument. Then the app will read the inputs from the file and then call the solver with the inputs. But I can’t get Julia to read a file. I’ve tried include and a macro that copy-and-pastes the file contents into the code (see below)with include since it puts the variable in a global scope or with eval since the name of the file is not known at compile time.

If you have a better method for dealing with input parameters please share!

Here’s a MWE showing what I’m trying to do based on ideas from julia - What exactly does include do? - Stack Overflow and `@def` macro generator broken on master - #3 by ChrisRackauckas

inputs.jl (Inputs File)

p = (
    a = 1,
    b = [(t) -> t^2]
)

main.jl (reads inputs and runs solver)

module Tester

# Macro to copy/paste file contents 
macro include(filename::AbstractString)
    path = joinpath(dirname(String(__source__.file)), String(filename))
    return esc(Meta.parse("quote; " * read(path, String) * "\n; end").args[1])
end

# Load parameters from file and call solver
function main(filename)
    include(filename)      # Include fails for function p.b
    #@include "inputs.jl"  # Macro works - but need to hard code filename
    #@include filename     # Macro fails with variable filename

    # Run solver with input parameters
    solver(p)
    return nothing
end

function solver(p)
    println("Running solver with p = ",p)
    println("a = ",p.a)             # Should be 1
    println("p.b[1] = ",p.b[1])
    fun = string_to_funct   
    println("b[1](3) = ",Base.invokelatest(p.b[1]))(3) # Should be 3^2=9
    return nothing
end

main("inputs.jl")
end

Your code using include already works for me, if I correct the brackets in invokelatest to Base.invokelatest(p.b[1], 3) and remove the string_to_funct line. To create an app, you should of course also move the “inputs.jl” argument into ARGS.

Full program
Folder structure
Tester
│   Manifest.toml
│   precompile_app.jl
│   Project.toml
│
├───compiled
│   ├───bin
│   │       julia.exe
│   │       ...
│   │       Tester.exe
│   │
│   └───share
│       └───julia
│               cert.pem
│               LocalPreferences.toml
│               Project.toml
│
├───inputs
│       inputs.jl
│       inputs2.jl
│
└───src
        Tester.jl

Most of these files are created by Pkg

using Pkg
Pkg.generate("Tester")

and PackageCompiler (see below).

Manually added files are Tester.jl, which corresponds to your main.jl,

Tester.jl
module Tester

# Load parameters from file and call solver
module Tester

# Load parameters from file and call solver
function julia_main()::Cint
    include(ARGS[1])  # Loads in p
	
    # Run solver with input parameters
    solver(p)
    return 0
end

function solver(p)
    println("Running solver with p = ",p)
    println("a = ",p.a)  # 1 for inputs1.jl, 3 for inputs2.jl
    println("p.b[1] = ", p.b[1])
    println("b[1](3) = ", Base.invokelatest(p.b[1], 3)) # 3^2 = 9 for inputs1.jl, 3^3 = 27 for inputs2.jl
end

end

the input files

inputs1.jl
p = (
    a = 1,
    b = [(t) -> t^2]
)

which is your inputs.jl

inputs2.jl
p = (
    a = 3,
    b = [(t) -> t^3]
)

and the optional precompilation script precompile_app.jl, which is not really useful for this small example, but could be for your full program.

precompile_app.jl
using Tester

push!(ARGS, "./inputs/inputs1.jl")
Tester.julia_main()

(cf. PackageCompiler.jl/examples/MyApp/precompile_app.jl at master · JuliaLang/PackageCompiler.jl · GitHub and PackageCompiler.jl how to specify ARGS for create_app? - #2 by jsjie)

To compile, I opened Julia in the parent directory of Tester, and used

]activate Tester
using PackageCompiler
create_app("Tester", "Tester/compiled")  
# or: create_app("Tester", "Tester/compiled", precompile_execution_file="Tester/precompile_app.jl")

(assuming ./Tester/compiled does not already exist). As output I get

> .\Tester\compiled\bin\Tester.exe .\Tester\inputs\inputs1.jl
Running solver with p = (a = 1, b = [Tester.var"#1#2"()])
a = 1
p.b[1] = #1
b[1](3) = 9
> .\Tester\compiled\bin\Tester.exe .\Tester\inputs\inputs2.jl
Running solver with p = (a = 3, b = [Tester.var"#1#2"()])
a = 3
p.b[1] = #1
b[1](3) = 27

You will notice that this is on Windows. If relevant, here’s my full versioninfo():

versioninfo
Julia Version 1.10.4
Commit 48d4fd4843 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 8 default, 0 interactive, 4 GC (on 8 virtual cores)
Environment:
  JULIA_NUM_THREADS = auto

and I am using PackageCompiler v2.1.17.


I kind of like the include approach you suggest, due to the amount of freedom it brings (you can just ‘import’ any object produced by Julia code), but I cannot say if there are any downsides, or what the idiomatic approach would be.

@eldee Thank you so much for your detailed answer. It really helped!

The only thing that wasn’t transferable was calling invokelatest within the solver. This requires significant modification of the solver and overhead each time the function is evaluated. To fix this I added code that makes a copy of the input parameters and calls invokelatest on any functions within the parameters. Then the solver is called and no modification of the solver is required.

Here’s the updated version of your Tester.jl that includes the function process_command_line_args(ARGS) that copies the NamedTuple of inputs and uses invokelatest as needed. It works for the types of inputs my program uses but could possibly be generalized to work on any type of inputs (scalar, vector, array of Ints, Floats, or Functions)

The function process_var(var) is ugly and limited to functions with 6 inputs. This is where invokelatest is applied if the input is a function. Any ideas on how to write this for any number of inputs to the function would be appreciated.

Tester.jl
# Load parameters from file and call solver
module Tester

# Load parameters from file and call solver
function julia_main()::Cint

    # Load and process input file
    p = process_command_line_args(ARGS)

    # Run solver with parameters
    solver(p)
    return 0
end

"""
Reads a file (ARGS[1]), extracts p, and processes p 
"""
function process_command_line_args(ARGS)
    
    # Check ARGS 
    length(ARGS) >= 1 || error("Provide name of paramater file as command line argument e.g. `>> julia inputs.jl`")

    # Assume filename is the first arguement 
    filename = ARGS[1] 

    # Load file containing parameters
    include(filename)

    # Check file loaded parameter variable p 
    (@isdefined p) || error("Parameter file should contain a NamedTuple with name `p`")
    (p isa NamedTuple) || error("Parameter file contains p, but p is not a NamedTuple")

    # Copy variables into new tuple and process functions
    d = NamedTuple()
    for (key,val) in zip(keys(p), p)
        # Check if this parameter contains a vector
        if val isa Vector 
            # Process vector entries
            val_vec = Any[]
            for n in eachindex(val)
                # Process each entry in the vector
                push!(val_vec,process_var(val[n]))
            end
            # Copy entries into d 
            d = merge(d, NamedTuple{(key,)}((val_vec,)))
        else 
            # Process value and copy into d
            d = merge(d, NamedTuple{(key,)}((process_var(val),)))
        end
    end
    return d
end

"""
Takes a variable, and makes functions executable
"""
function process_var(var)
    if var isa Function
        # Determine number of function arguments 
        m = first(methods(var))
        n = m.nargs - 1 # remove 1 because includes function name as 1st arg
        if     n==0; return (           ) -> Base.invokelatest(var,            )
        elseif n==1; return (a          ) -> Base.invokelatest(var, a          )
        elseif n==2; return (a,b        ) -> Base.invokelatest(var, a,b        )
        elseif n==3; return (a,b,c      ) -> Base.invokelatest(var, a,b,c      )
        elseif n==4; return (a,b,c,d    ) -> Base.invokelatest(var, a,b,c,d    )
        elseif n==5; return (a,b,c,d,e  ) -> Base.invokelatest(var, a,b,c,d,e  )
        elseif n==6; return (a,b,c,d,e,f) -> Base.invokelatest(var, a,b,c,d,e,f)
        else; error("Processing a function with $n arguments is not programmed")
        end
    else
        return var
    end
end

function solver(p)
    println("\nRunning Solver...")
    # Testing p.a
    @show p.a  # 1 for inputs1.jl, 3 for inputs2.jl
    # Testing p.b
    @show p.b[1](3)
    @show p.b[1](4)
    # Testing p.c
    @show p.c[1](3,4)
    @show p.c[1](4,5)
    @show p.c[2](3,4)
    @show p.c[2](4,5)
    # Testing p.d
    @show p.d()
    # Testing p.e 
    @show p.e(1,2,3,4,5)
    # Testing p.f 
    @show p.f

end

end

with input files with more types of inputs

inputs1.jl
p = (
    a = 1,
    b = [(t) -> t^2],
    c = [(x,y) -> x^2+y^2,
         (z,w) -> z + w  ],
    d = () -> 5,
    e = (a,b,c,d,e) -> 6+e,
    f = [1 2 3
         4 5 6],
)
inputs2.jl
p = (
    a = 3,
    b = [(t) -> t^3],
    c = [(x,y) -> x^2+y^2,
         (z,w) -> z + w  ],
    d = () -> 15,
    e = (a,b,c,d,e) -> 6+a,
    f = [10 20 30
         40 50 60],

)

Glad I could help!

You can use splatting:

"""
Takes a variable, and makes functions executable
"""
function process_var(var)
    if var isa Function
        _func(x...) = Base.invokelatest(var, x...)
        return _func
    else
        return var
    end
end


By the way, if you want to get rid of the downside of the include approach that it pollutes the global scope, here’s an @include macro which works for me.

macro include(filename)
    return quote
        file_contents = read($filename, String)
        load_expr = Meta.parse(file_contents)
        return Meta.eval(load_expr)
    end |> esc
end

This is compatible with the input files as they are (as e.g. Meta.eval(:(t = 1)) returns 1), but to me it seems a bit cleaner if you remove the p = (in the example Meta.eval(:1) clearly also returns 1).

inputs1.jl
(
    a = 1,
    b = [(t) -> t^2],
    c = [(x,y) -> x^2+y^2,
         (z,w) -> z + w  ],
    d = () -> 5,
    e = (a,b,c,d,e) -> 6+e,
    f = [1 2 3
         4 5 6],
)

In Tester.jl’s process_command_line_args, you’d then use

    # Load file containing parameters
	p = @include filename

    # Check file contained variables in a NamedTuple
    (p isa NamedTuple) || error("Parameter file does not contain a NamedTuple")

(Here the assignment of the @include return is required, but you can freely choose the variable name. This also means that p (or whichever variable you chose) will always be defined (assuming we could didn’t error while parsing the inputs file), so we no longer need the @isdefined.).

Well, any include-like approach can open the door to malicious code insertion, if a bad actor would alter the inputs file, so that’s a major downside :slight_smile: .

1 Like