I’m working on a package that allows users to define pipeline tasks in an ergonomic way and then schedules jobs and their dependencies with
I have a macro that takes a user input like this:
@Job begin name = MyJob parameters = (a,b) dependencies = AnotherJob(a) process = f(a,b) end
The macro then takes this information, makes a new struct called
MyJob and splices the user’s input into to
run_process(j::MyJob), and a few others. My test suite works perfectly from the Pkg test environment.
The macro's code looks like this:
""" Unpack the parameters of the job, then splice the body of the function provided by the user """ function unpack_input_function(function_name, job_name, parameter_names, function_body, additional_inputs=()) return quote function Waluigi.$function_name(job::$job_name, $(additional_inputs...)) let $((:($name = job.$name) for name in parameter_names)...) $function_body end end end end macro Job(job_description) # This returns a Dict of job parameters and their values job_features = extract_job_features(job_description) #Truncating some complicated code here that just extracts the parameters paramter_names = complicated_code(job_features) job_name = job_features[:name] dependency_ex = unpack_input_function(:get_dependencies, job_name, parameter_names, job_features[:dependencies]) process_ex = unpack_input_function(:run_process, job_name, parameter_names, job_features[:process]) struct_def = :(struct $job_name <: AbstractJob end) push!(struct_def.args.args, parameter_names...) return quote $struct_def eval($(esc(dependency_ex))) eval($(esc(process_ex))) end end
However, I just tried to add the package to another project. When I try to define a Job, I get the following error when trying to precompile:
LoadError: Evaluation into the closed module `Waluigi` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Waluigi` with `eval` during precompilation - don't do this.
I think I understand that adding new type definitions and/or function dispatches within the dependency module doesn’t work, but I’m not sure how to approach changing the way the macro defines things to get it to work properly. By that I mean, I want the generic pipeline code I’ve written that calls
get_dependencies to see the new dispatches.
Or it’s possible I’m approaching this in a completely wrong way. Open to that feedback too