Loading arbitrary user code in a package (including imports)

General Usage
1

I don’t really see a way how this is possible, but just to make sure that I’m not missing anything – I have a package with the following function, which takes a user-supplied .jl file and includes it in an ad-hoc module:

function read_input(path)
	mod = @eval baremodule $(gensym()) end
	input = Base.include(mod, path)
	println(input)
end

While this works, the user-supplied code can only import modules which are listed in the package’s Project.toml. For other modules, the result is a LoadError:

ERROR: LoadError: ArgumentError: Package TestPkg does not have SpecialFunctions in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have TestPkg checked out for development and have
  added SpecialFunctions as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with TestPkg
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1776 [inlined]
  [2] macro expansion
    @ ./lock.jl:267 [inlined]
  [3] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
  [4] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
  [5] invoke_in_world
    @ ./essentials.jl:923 [inlined]
  [6] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
  [7] include
    @ ./Base.jl:495 [inlined]
  [8] read_input(path::String)
    @ TestPkg ~/loading_test/TestPkg.jl/src/TestPkg.jl:5

Is there any way to do this? I imagine that even if it is, this would probably mess with precompilation, so it’s probably not worth it?

For some larger context, this is the old use case of setting up inputs and parameters for numerical simulations again. The .jl file to be read is code that creates these inputs, and returns them in the expected format as the value of its expression. I know that there are tons of different ways to do this, but in the end, code is the most flexible since it doesn’t have any limitations on what can be done. It’d be nice to avoid having to write these things to a file, only to immediately have to read it in again.

As to why I’m doing it this way around and not just have the user write a script where they call the relevant function from the package, there is also a main “executable” (script) with a command-line interface that would be circumvented if the user input was also the main entry point. In other words, while there should be flexibility in defining the simulation inputs, I’m trying to reduce the need for code as input to a minimum.

2

One way to do this is to have the user script call the main CLI function at the end of the setup script. This is what BinaryBuilder does, for example see: Yggdrasil/A/ABC/build_tarballs.jl at master · JuliaPackaging/Yggdrasil · GitHub
and
MEDYANSimRunner.jl/test/example/main.jl at main · medyan-dev/MEDYANSimRunner.jl · GitHub

The ARGS gets passed to build_tarballs function after all the setup work is done, so you can still have a nice CLI.

Another option would be to have the user also provide Manifest and Project toml files, and then evaluate the input script in the provided environment using Malt.jl, but this can get messy.

3

Thanks! What I didn’t like with the “user script calls CLI” option is that it adds boilerplate. I was about to say that there’s no way to guarantee that this call is done, but I guess it is possible to throw an error if the function has not been called.

Thanks for the pointer to Malt.jl! I thought about creating a separate Julia process, but didn’t realize that there’s something ready-made that takes care of all the serialization business. I don’t think the environment would be such a problem, though? If the user wants to use a specific environment, they can always activate that within their input code file.

4

I’d expect an ad-hoc module intending to evaluate my code to be itself evaluated into my module or REPL, not into an imported package. That should reach the user’s environment instead. Is this not possible for the use case?

1 Like
5

Well, the intended user interface is the CLI, so there’s no user module or REPL in that case. But maybe there’s a misunderstanding – the ad-hoc module is the one I’m creating via @eval baremodule $(gensym()) end, and its only purpose is to be the module targeted by include(), since the point is not actually any global state, but the value returned from using include() on the user’s .jl file.

6

My current version of a solution is something like

worker = Malt.Worker()
Malt.remote_eval_wait(worker, quote
	using Pkg
	Pkg.activate($(Base.active_project()); io=devnull)
end)
input = Malt.remote_eval_fetch(worker, :(include($path)))
Malt.stop(worker)

Thanks again @nhz2 for mentioning Malt.jl!

7

Any Julia process has a Main module at least, as julia -e 'println(@__MODULE__)' demonstrates. What I’m saying is the ad hoc module could be evaluated into a module stemming from the user’s process instead of your package. For an example off the top of my head, albeit more convoluted, RuntimeGeneratedFunctions.jl initializes a cache and underlying generated function method in the user’s module, and the user is given a macro to construct RuntimeGeneratedFunction instances for expressions that default to the user’s module by expanding @__MODULE__.

1 Like
9

Ah, I see what you’re saying now – you mean evaluating into a different module in the line @eval baremodule $(gensym()) end. I got sidetracked by the talk about a “user’s module”, which doesn’t exist in this case (the julia process is started via the CLI script, which defines the Main module and is also written by me). I didn’t realize that this was what’s making the difference!

In fact, evalfile (mentioned by @nsajko – not sure why the post is deleted now?) seems to do exactly what I want!

1 Like
10

I had to reword myself because “user’s module” reasonably but incorrectly implied the user manually wrote module ... end somewhere. As long as the CLI script is executed by the user, then its Main has the user’s environment, not your package’s separate environment.

1 Like