# Julia-based Makefile replacement for research workflows
**URL:** https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368
**Category:** General Usage
**Tags:** question, workflow
**Created:** [January 10, 2022, 11:40pm UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368 "2022-01-10T23:40:56Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@goerz](https://discourse.julialang.org/u/goerz)
#### Post date: [January 10, 2022, 11:40pm UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/1 "2022-01-10T23:40:56Z")
I’m playing around with [DrWatson](https://github.com/JuliaDynamics/DrWatson.jl) to organize research workflows. It feels like it would be nice to also have a `Makefile` in a project repository that defines which scripts produce which output file. The idea would be to simply be able to run `make` in the project root and that would run all my julia scripts to produce output files, plots, and reports.
As far as I can tell, this is not something that DrWatson has built-in: it can keep track of which script (and which revision) generated a particular output file, but not really the inverse direction, and without any good way to automate the entire project.
One problem with Makefiles that I ran into was that, by default, DrWatson [likes to put equal signs in the output files](https://juliadynamics.github.io/DrWatson.jl/dev/name/#DrWatson.savename), which [Makefiles have problems with](https://stackoverflow.com/questions/70659579/equal-sign-in-makefile-target). Thus (apart from ugly workarounds), I’d either have to [convince DrWatson not to do that](https://github.com/JuliaDynamics/DrWatson.jl/issues/314), or switch to something other than a `Makefile`.
Is there some package in Julia that can replace `make` for these kinds of workflows? Something that takes a data structure defining a map from a target to one or more dependencies, and runs a particular piece of Julia code to create/update the target if it doesn’t exist or is older than the dependency?
This doesn’t have to be a _full_ replacement of `make` suitable for driving software compilation. Something that can emulate a relatively simple `Makefile` rule like
```julia
plot.pdf: plot.jl data/*.dat
julia $< # equivalent to `julia plot.jl`
```
would be sufficient. I could probably code up something quick and dirty in any particular project, but maybe there is an existing solution? I saw [Alternatives to Makefile and shebang scripts?](https://discourse.julialang.org/t/alternatives-to-makefile-and-shebang-scripts/7628), but all projects mentioned there seem to be abandoned.
A huge added benefit of using Julia for this would also be that the entire “make” can happen in a single julia process, reducing the JIT overhead. Also, it would work on non-Unix systems.
---
### Author:  [@goerz](https://discourse.julialang.org/u/goerz)
#### Post date: [January 10, 2022, 11:47pm UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/2 "2022-01-10T23:47:18Z")
Associated issue: [https://github.com/JuliaDynamics/DrWatson.jl/issues/315](https://github.com/JuliaDynamics/DrWatson.jl/issues/315)
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [January 11, 2022, 2:50am UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/3 "2022-01-11T02:50:07Z")
> [@\[ANN\] Makeitso.jl: Make-like behaviour for Julia scripts](https://discourse.julialang.org/t/ann-makeitso-jl-make-like-behaviour-for-julia-scripts/18243):
>
> Makeitso.jl Make-like behavior to facilitate writing long-running scripts. The idea is that the user supplies the dependencies between targets and recipes for how to make a target from those dependencies. If changes happen to one recipe (cf. if one edits one source file in a make project), only affected variables will be recomputed. Upon computation, a backup is written to disk. This means that even when the work on the script is resumed in a different session (because you went home, julia cr…
---
### Author:  [@goerz](https://discourse.julialang.org/u/goerz)
#### Post date: [January 11, 2022, 6:29am UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/4 "2022-01-11T06:29:36Z")
Hmm… `Makeitso.jl` looks like an interesting project in terms of tracking a full dependency graph, but it seems pretty far removed from anything file-based like `make`.
In the meantime, “quick and dirty” would look something like [this `src/makerules.jl` file](https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints/blob/master/src/makerules.jl), which is then included in a [`make.jl` file](https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints/blob/master/make.jl) in the project root (and which in turn can be run from a traditional [`Makefile`](https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints/blob/master/Makefile)). The `AbstractRule`/`ScriptRule` should be easy to extend to more complicated situations. This doesn’t do anything like recursive dependencies. The assumption is that the `RULES` are ordered by hand so that all prerequisites exist by the time the rule is evaluated. That’s where something like `Makeitso.jl` would come in, presumably, but I’m not sure it’s worth the trouble.
---
### Author:  [@krcools](https://discourse.julialang.org/u/krcools)
#### Post date: [January 11, 2022, 10:32am UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/5 "2022-01-11T10:32:53Z")
Makeitso is very much file based in that every target is cached on disk. What you probably mean is the ability to produce source and object files as part of a compilation process as in make. I guess this can be easily achieved in Makeitso.jl by creating those output files as a side effect of the recipe. You would have two files on disk then: one containing the payload and another pilot file that keeps track of the recipe hash and the completion timestamp.
I will give it some though how this can be streamlined (perhaps a decicated macro?).
What I do think is missing in Makeitso.jl is the ability - like in the amazing Pluto.jl - to have the dependency graph automatically created and maintained. But contrary to Pluto, which is reactive, Makeitso or any make-like tool should be imperative. Expensive to build targets are only generated on explicit request of the target or its dependant targets.
---
### Author:  [@goerz](https://discourse.julialang.org/u/goerz)
#### Post date: [January 11, 2022, 11:46am UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/6 "2022-01-11T11:46:25Z")
> [@krcools](#):
>
> Makeitso is very much file based in that every target is cached on disk. What you probably mean is the ability to produce source and object files as part of a compilation process as in make.
Yes, that’s exactly what I mean. Sorry I wasn’t clear.
> I guess this can be easily achieved in Makeitso.jl by creating those output files as a side effect of the recipe.
Right, it would certainly be possible to use Makeitso.jl as a “backend” in the code I have in [`makerules.jl`](https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints/blob/master/src/makerules.jl)/[`make.jl`](https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints/blob/master/make.jl). Specially, it would allow discarding the manual order of the rules that I have now, and instead have Makeitso.jl figure out the order based on missing/outdated dependencies, exactly like `make` does it.
> You would have two files on disk then: one containing the payload and another pilot file that keeps track of the recipe hash and the completion timestamp.
That actually seems less than ideal, in this context. I don’t know if there would be a way to tweak Makeitso.jl to use the `mtime` of the “payload file” for the time stamp
---
### Author:  [@krcools](https://discourse.julialang.org/u/krcools)
#### Post date: [January 12, 2022, 1:34pm UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/7 "2022-01-12T13:34:26Z")
> [@goerz](#):
>
> That actually seems less than ideal, in this context. I don’t know if there would be a way to tweak Makeitso.jl to use the `mtime` of the “payload file” for the time stamp
I think there are two reasons not to get rid of the pilot files. (i) If produced variables are still in memory, the timestamp does not require poking files on disk (I know this is not of particular concern in the use case of interest to you), and (ii) the pilot file also stores the recipe hash so it can track invalidation of not only the dependent targets but also of the build script itself.
---
### Author:  [@martinmestre](https://discourse.julialang.org/u/martinmestre)
#### Post date: [September 3, 2024, 1:17pm UTC](https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/8 "2024-09-03T13:17:45Z")
Hello all, have you found a solution? I am about to start using ShowYourWork for the research pipeline (and paper writing) but wanted to know if there is any solution in Julia.