I’m playing around with DrWatson to organize research workflows. It feels like it would be nice to also have a
Makefile in a project repository that defines which scripts produce which output file. The idea would be to simply be able to run
make in the project root and that would run all my julia scripts to produce output files, plots, and reports.
As far as I can tell, this is not something that DrWatson has built-in: it can keep track of which script (and which revision) generated a particular output file, but not really the inverse direction, and without any good way to automate the entire project.
One problem with Makefiles that I ran into was that, by default, DrWatson likes to put equal signs in the output files, which Makefiles have problems with. Thus (apart from ugly workarounds), I’d either have to convince DrWatson not to do that, or switch to something other than a
Is there some package in Julia that can replace
make for these kinds of workflows? Something that takes a data structure defining a map from a target to one or more dependencies, and runs a particular piece of Julia code to create/update the target if it doesn’t exist or is older than the dependency?
This doesn’t have to be a full replacement of
make suitable for driving software compilation. Something that can emulate a relatively simple
Makefile rule like
plot.pdf: plot.jl data/*.dat julia $< # equivalent to `julia plot.jl`
would be sufficient. I could probably code up something quick and dirty in any particular project, but maybe there is an existing solution? I saw Alternatives to Makefile and shebang scripts?, but all projects mentioned there seem to be abandoned.
A huge added benefit of using Julia for this would also be that the entire “make” can happen in a single julia process, reducing the JIT overhead. Also, it would work on non-Unix systems.