Using GNU Make, DrWatson, and Julia scripts to track dependencies and automate script running

General Usage
, ,
1

I have a number of scripts that I am trying to run using GNU Make, but within them, I am using DrWatson’s @produce_or_load macro to save parameter combinations to different files. I would like to use GNU Make to track dependencies and re-run later scripts when ones higher in the dependency tree have been updated. My understanding of Make is that it requires the output of a Make command to be a file so it can compare file modification times to calculate what needs to be re-run. If I use DrWatson to save the outputs instead of Make, is there a good way to have Make track dependencies? Currently I am using touch to create an empty temporary file with the same name as the script file, but this feels a bit hacky, and doesn’t track the actual outputs. It also seems to make it hard to use Make automatic variables etc (although, I am very new to Make, so I could just be missing something).

In the below mock example, I’m just using JLD2 to directly save (and load) the output, but it’s the same concept as using @produce_or_load in more complicated situations later - the saving and loading happens within the Julia scripts.

Makefile

.PHONY: all
all: tmp/single-sim_setup tmp/single-sim

tmp/single-sim_setup: src/single-sim_setup.jl
	julia $^
	@touch $@

tmp/single-sim: src/single-sim.jl tmp/single-sim_setup
	julia $^
	@touch $@

.PHONY: clean
clean:
    rm -rf tmp/*
    rm -rf data/singlesim/*

src/single-sim_setup.jl

using DrWatson
@quickactivate "TestCode"

using JLD2

states = [100, 200, 300]
time = 0:1:100

jldsave("data/singlesim/single-sim_setup.jld2"; states, time)

scripts/single-sim.jl

#%%
using DrWatson
@quickactivate "TestCode"

using UnPack
using JLD2
using Distributions
using DataFrames

#%%
@unpack states, time = load("data/singlesim/single-sim_setup.jld2")

#%%
out = map(mean -> rand(Normal(mean), length(time), states)
out_df = DataFrame(out, :auto)

#%%
jldsave("data/singlesim/single-sim_arrays.jld2"; out, out_df)

I originally used lots of chained include() statements between scripts, but that made it difficult to develop multiple scripts that depended on the same source data, but had different requirements and tasks. I also wanted to make sure that everything was re-run when the dependencies were updated, and it was getting hard to keep track of which scripts to re-run.

If there is a better way of doing this (either with or without Make), please let me know. And if you need more information, I would be happy to expand upon anything.