Loading custom models: dynamic code evaluation?

General Usage
1

I’m trying to implement the following or similar architecture, and would really like to hear some advice on how to do it correctly/better/…
Basically, there is going to be many user-defined models, which I think are reasonable to organize like this:

# file model_A.jl
struct ModelParameters
 A::Float64
end

eval(m::ModelParameters, x) = ...
loglike(m::ModelParameters, x, y) = ...

# file model_B.jl
struct ModelParameters
 B::Float64
end

eval(m::ModelParameters, x) = ...
loglike(m::ModelParameters, x, y) = ...

Then there will be a common file, that handles data loading and e.g. fitting:

function load_model(path)
    global model_path = path
    include(path)
end

function load_data()
    global data = ...
end

function fit_model()
    ...
    loglike(ModelParameters(...), data.x, data.y)
    ...
end

load_model("model_A.jl")
fit_model()

so far so good. But what to do when I need several models together with their parameter classes and all methods loaded at the same time e.g. for comparison?

What do you thing of this approach in general? Anything obviously wrong, or something better exists?

2

No, I definitely wouldn’t recommend this approach. You’ve correctly identified the primary problem, which is that there is no way for both of your files to coexist if they try to declare the same types.

Instead, consider creating a module just for the interface you want your models to create, and then any of your various models can implement that common interface in whatever way they see fit. For example:

module ModelInterface
  export loglike

  # We don't have to add any methods to these functions, yet. Instead
  # we're just declaring that they exist inside ModelInterface so that
  # we can add methods later.
  function loglike end
end

module ModelA
  struct ModelParameters
    A::Float64
  end
  ModelInterface.loglike(m::ModelParameters, x) = ...
end

module ModelB
  struct ModelParameters
    B::Float64
  end
  ModelInterface.loglike(m::ModelParameters, x) = ...
end

And then your common code might look something like:

using ModelInterface

function fit_model(model)
  loglike(model, ...)
end

In this way you as the user would choose which kind of model to use simply by passing it into fit_model:

fit_model(ModelA.ModelParameters(...))
fit_model(ModelB.ModelParameters(...))

Of course, you can still choose to make one of those types the default if you want.

The important point is that although both of your ModelParameters structs happen to have the same name, they live in different modules and can therefore coexist without any issues or confusion. But because they both implement the same loglike function from the ModelInterface module, they can be used interchangeably.

1 Like
3

I would separate all the shared behaviours and functions into a package, and do the actual fitting in a script project. Then the parent package can be useful to other people if it works out well.

I would also give all the models different names, and inherit from an abstract type.

In the parent package:

module ModellingPackage

export AbstractModelParameters, eval, loglike, fit_model

abstract type AbstractModelParameters

# Default behaviours
eval(m::AbstractModellingParameters, x) = ...
loglike(m::AbstractModellingParameters, x, y) = ...

function load_model(path)
    global model_path = path
    include(path)
end

function load_data()
    global data = ...
end

function fit_model(model)
    ...
    loglike(model, data.x, data.y)
    ...
end

end

In user defined code:

# file model_A.jl
struct AParameters <: AbstractModelParameters
 A::Float64
end

eval(m::AParameters, x) = ...
loglike(m::AParameters, x, y) = ...

# file model_B.jl
struct BParameters
 B::Float64
end
eval(m::BParameters, x) = ...

User models can inherit behaviours from AbstractModelParameters where it makes sense. Here BParameters just uses the default loglike method.

Then load user modules in the script project:

using ModellingPackage
import ModellingPackage: eval, loglike

include("user/model_A.jl")
include("user/model_B.jl")

load_params(AParameters)
fit_model(AParameters(params...))

load_params(AParameters)
fit_model(BParameters(params...))

I normally also use Parameters.jl or my Defaults.jl package to set default params so the models can run without the file load.

using Paramers
@with_kw struct AParameters <: AbstractModelParameters
 A::Float64 = 1.75
end

or

using Defaults
@default_kw struct AParameters <: AbstractModelParameters
 A::Float64 | 1.75
end
4

Is it possible to specify several models dynamically by their paths in this case? Something along these lines:

module ModelInterface
  function loglike end
  export loglike
end

paths = ["model_a.jl", "model_b.jl", "model_c.jl"]
modtypes = []
for p in paths
  modname = "Model_$(splitext(basename(p))[1])"
  @eval :(module $modname
    include($p)
  end)
  push!(modtypes, :($modname.ModelParameters))
end

ModelInterface.loglike.(modtypes, x)  # e.g. plot or compare models

but working and less hacky?

5

As I understand your suggested approach, it has certain difficulties with dynamic loading of models which are specified by their source paths. If each model has different type name, then one has to know the type name when includeing the model file in order to run some of its functions. Also (less importantly) it duplicates the model name in many places, instead of just the filename.

6

Sound like you have a lot of models to compare… But I don’t quite understand the problem

If they are all subtypes of AbstractModelParameters you can just load all the files, then:

for modeltype in subtypes(AbstractModelParameters)
    fit_model(modeltype(loadparams(modeltype)...))
end