"LoadError: Evaluation into the closed module" with Macro that generates new types and methods

General Usage
1

I’m working on a package that allows users to define pipeline tasks in an ergonomic way and then schedules jobs and their dependencies with Dagger.jl.

I have a macro that takes a user input like this:

@Job begin
    name = MyJob
    parameters = (a,b)
    dependencies = AnotherJob(a)
    process = f(a,b)
end

The macro then takes this information, makes a new struct called MyJob and splices the user’s input into to get_dependencies(j::MyJob), run_process(j::MyJob), and a few others. My test suite works perfectly from the Pkg test environment.

The macro's code looks like this: 
"""
Unpack the parameters of the job, then splice the body of the function provided by the user
"""
function unpack_input_function(function_name, job_name, parameter_names, function_body, additional_inputs=())
    return quote
        function Waluigi.$function_name(job::$job_name, $(additional_inputs...))
            let $((:($name = job.$name) for name in parameter_names)...)
                $function_body
            end
        end
    end
end

macro Job(job_description)
    # This returns a Dict of job parameters and their values
    job_features = extract_job_features(job_description)
   
   #Truncating some complicated code here that just extracts the parameters
    paramter_names  = complicated_code(job_features)
    job_name = job_features[:name]
    
    dependency_ex = unpack_input_function(:get_dependencies, job_name, parameter_names, job_features[:dependencies])
    process_ex = unpack_input_function(:run_process, job_name, parameter_names, job_features[:process])
    
    struct_def = :(struct $job_name <: AbstractJob end)
    push!(struct_def.args[3].args, parameter_names...)
    return quote
        $struct_def
        eval($(esc(dependency_ex)))
        eval($(esc(process_ex)))
    end
end

However, I just tried to add the package to another project. When I try to define a Job, I get the following error when trying to precompile:

LoadError: Evaluation into the closed module `Waluigi` breaks incremental compilation because the side
effects will not be permanent. This is likely due to some other module mutating `Waluigi` with `eval` 
during precompilation - don't do this.

I think I understand that adding new type definitions and/or function dispatches within the dependency module doesn’t work, but I’m not sure how to approach changing the way the macro defines things to get it to work properly. By that I mean, I want the generic pipeline code I’ve written that calls get_dependencies to see the new dispatches.

Or it’s possible I’m approaching this in a completely wrong way. Open to that feedback too :slight_smile:

2

FWIW: This post might be the solution, but since it was using eval in a function, I’m not sure how to apply it to my situation.

3

A macro already returns code that is then evaluated - using eval in a macro is likely going to be the issue here (even though the eval is part of the returned expression, there’s no need for it). Do you have an example of the code you would have written/that you want the macro to expand to? Also, what does your macro return without those eval (use @macroexpand to find out without actually running the resulting code)? Do the two match?

1 Like
4

Yes, @macroexpand matches my expected example.

Truncating a bit:

struct MyJob <: Waluigi.AbstractTarget
    a
    b
end

function Waluigi.get_dependencies(job::MyJob)
    let a = job.a, b = job.b
        AnotherJob(a)
    end
end

function Waluigi.run_process(job::MyJob, dependencies)
    let a = job.a, b = job.b
        f(a, b)
    end
end

You’re right that, because I added eval each of the function definitions, they are wrapped in Walugi.eval(...). However, if I don’t do this, then when I later try to run the code that calls run_process, it doesn’t find the methods that the macro defines. If I add the eval, it works.

After writing all this out, I’m realizing my actual question is:

While still evaluating the definition of a method in the user’s module space, how do I get my package to pick up new the methods? When I want to extend a function from Base, it’s sufficient to just add the source module’s name Base.sum(::MyType) = ..., and everything seems to pick up on it.

5

What error message do you get? I suspect your use of just putting esc on the whole returned expression and not on something like Waluigi.run_process ends up breaking hygiene. I can’t verify that though because your MWE is not complete - I don’t have access to extract_job_features, for example.

1 Like
6

Ah, thank you so much, @Sukera! I was being too broad with excing

Here is an actually working MWE for what I was trying to do for future googlers:

module A
abstract type Abstract end
f(::Abstract) = error("Not Implemented")

macro make_type(type_name)
    quote
        struct $type_name <: $(esc(A.Abstract)) end
        function A.f(::$(esc(type_name)))
            println("It worked!")
        end
    end
end

end # mod A

using .A 

module B
using ..A

A.@make_type NewType

A.f(NewType())
# prints "it works"!
end