Dynamic Imports and User Defined Interfaces in Julia

New to Julia
, , ,
1

Hello! I am a newcomer to Julia and I am trying to recreate something that I worked a bit on in Python in Julia, because I think Julia will be the better language overall for numerical performance. However, I am trying to do something that I am not sure Julia is suited for before I can get to that numerical aspect.

The code needs to interact with several outside sources of data, like databases, web APIs, etc. What I would like is for a configuration file to be able to define which files to import for each of these sources. Then, the program will import those source files and use the functions within to download data, store it away, etc. In Python I would do this with an abstract base class, and dynamically import the module based on the name saved in the configuration file. In Julia, I am not so sure. It seems there are a few options possible with metaprogramming, but I am not sure they fit exactly into what I imagine. Maybe something like the following as a minimum (but not working!) example would help:

# Main file
include("ConfigImport.jl")

function main()
    imported_file = ConfigImport.import_config() # where imported file is a reference to OtherFile
    imported_file.do_stuff()
end

main()

# ConfigImport.jl

module ConfigImport

using TOML

function import_config()

config_data = nothing
    try
        config_data = TOML.parsefile("config/config.toml")

    catch e
        if isa(e, SystemError)
            println("It appears the config file is missing!")
        elseif isa(e, TOML.ParserError)
            println("It appears there is something wrong with the format of the config file!")
        else
            throw(e)
        end
    end

    include(config_data["OtherFile"] * ".jl")
    return Symbol(config_data["OtherFile"])
end
end

# OtherFile.jl

module OtherFile

function do_stuff()
    println("Doing some stuff!")
end

end

I feel like there has to be a better design pattern or a use of macros to accomplish what I am trying to do, I just have no idea what that might be.

2

Don’t look for macros here, you need dynamic code loading, macros are for static code transformations.

I think that including only what you need may be OK, although I don’t see why exactly you need it instead of just including all the code and use the functions that are needed based on the conf.

If you really need the dynamic loading, I think going with .jl config files are much easier than using .toml or any other format. You can simply write the include statements directly to the .jl config file and let the compiler execute it instead of parsing and interpreting it manually.

For handling different environments, Configs.jl may also worth a look.

3 Likes
3

That’s not really the cleanest way of doing things in Julia. Julia has multiple dispatch - so you can just define the same method multiple times for different data types - and generate different data types depending on user config.

Basically it’s best to try to use simple methods and the type system before you use metaprogramming or some kind of code loading trick. You may want to do some reading and exploration of types. Direct transfer of python idioms will be a lot less successful than re-imagining the code in the context of having an amazing type system and multiple dispatch that python simply doesn’t have.

5 Likes
4

I don’t really need to have the code only load certain modules, so I think I will probably try to avoid doing so since it seems to be a lot of trouble. I will check out Configs.jl, since it seems like it could make things a bit easier. Thanks!

5

I was about to write something about functions that I wouldn’t expect to need to pass data into… and then I remembered that they have no saved state because they aren’t attached to objects, so of course I will need to pass the configuration data into the functions. With that in mind, I’m starting to wrap my brain around using multiple dispatch for something like this. Thanks!

2 Likes