I come from Python and I very often find myself loading code dynamically based on filesystem paths provided by the user (typically through command line). Typically they are either plugin code or configuration files (I like having my configurations as code). So each of these dynamically loaded files contains some “standard” methods / objects that are named the same in each file (which I then use from my main codebase) plus some internal components (which are typically file-specific and may or may not have name clashes).
In python I would do something like this:
import importlib
import os
import sys
plugins = []
for plugin_path in requested_plugins:
plugin_folder, plugin_name = os.path.split(plugin_path)
sys.path.append(plugin_folder)
plugin = importlib.import_module(plugin_name)
plugin.activate() # every plugin file defines an activate() function
plugins.append(plugin) # store every plugin to later call other functions defined by them
However, I don’t know how to achieve the same in Julia.
I could not find a way to make import and using work with dynamic file paths. include works fine with dynamic paths, but has many problems:
it imports everything into the global scope; if the various plugins define structures with the same name but conflicting definitions, Julia throws an error
because everything is imported in the standard scope, there’s no need to return (and probably no way in Julia to build) a reference to the imported file; therefore, there’s no way to store a list of enabled plugins and the “standardized” functions in the global scope are those from the last plugin loaded
if for some reason I end up loading the same plugin again, Julia will happily rerun the file, potentially causing all sorts of issues, while Python caches all imported files and always returns the same reference to them.
So, what is the best way to achieve in Julia a similar behaviour to that piece of Python code above?
I had thought about using modules before, but I couldn’t really understand how I would access the contents of each module without statically knowing the module name, which I don’t know because the loaded plugins will be different at each execution.
However, I now realized that I can use Julia’s metaprogramming capabilities to do that. So I came up with this code:
# Plugin loading phase
plugins = map(requested_plugins) do plugin_name
include("$plugin_name.jl")
Symbol(plugin_name)
end
# Unrelated code here...
do_other_unrelated_things()
# Plugin usage phase
for plugin in plugins
eval(:($plugin.foo()))
end
With this, I can store my plugins and call their functions whenever I need to without polluting the global scope. It’s not perfect (as I will explain now), but it’s good enough that I can use it without issues in my current project.
There are still two things that Python’s import_module does better:
with this Julia code, I need to ensure that each plugin defines a module with a different name. It’s something to keep in mind, but in my scenario it’s not a deal breaker and so I consider it only a minor issue.
with this code, there’s still the problem of potentially loading the same module multiple times, which leads at best to an annoying warning on the console, and at worst to wrong behaviour if some magic is going on in the modules. While I don’t plan any magic in my modules, I still find this slightly more annoying than issue 1. It’s not a problem if I call my code from the command line (I can dedup any plugins passed as command line arguments), but it becomes annoying if I load my code in a REPL and run it multiple times (Spawning a Julia interpreter seems awfully slow, so I’m using this second option a lot more than in other languages)
@patrick-kidger I hoped I could replace include with @from as you suggested, to fix issue 2. But it does not seem to work in my case, or maybe I’m just too inexperienced with Julia to make it work properly. The issue seems to be that I need to use a statically-known string as the path to the file. If I try to use string interpolation as I do in my code above with include, I get a MethodError.
So, after a few more experiments, I think I might settle on the following code:
# Plugin loading phase
plugins = []
map(requested_plugins) do plugin_name
plugin = Symbol(plugin_name)
if ! @eval @isdefined($plugin)
include("$plugin.jl")
end
push!(plugins, plugin)
end
# Unrelated code here...
do_other_unrelated_things()
# Plugin usage phase
for plugin in plugins
@eval $plugin.foo()
end
It puts symbolic names of all requested plugin modules in a vector for later use, but only includes the corresponding files if they were not already included (which may happen when using the REPL).
@patrick-kidger ahah! Thanks! I knew there must have been some way to do it that I did not know.
Ok, here’s the new code using that syntax. I also switched requested_plugins from plugin names to plugin paths, to make it more similar to my original Python code
using FromFile: @from
plugins = []
map(requested_plugins) do plugin_path
plugin_name = splitext(splitdir(plugin_path)[2])[1]
plugin = Symbol(plugin_name)
@eval @from $plugin_path import $plugin
@eval $plugin.activate()
push!(plugins, plugin)
end