I have a number of scripts that I am trying to run using GNU Make, but within them, I am using DrWatson’s
@produce_or_load macro to save parameter combinations to different files. I would like to use GNU Make to track dependencies and re-run later scripts when ones higher in the dependency tree have been updated. My understanding of Make is that it requires the output of a Make command to be a file so it can compare file modification times to calculate what needs to be re-run. If I use DrWatson to save the outputs instead of Make, is there a good way to have Make track dependencies? Currently I am using
touch to create an empty temporary file with the same name as the script file, but this feels a bit hacky, and doesn’t track the actual outputs. It also seems to make it hard to use Make automatic variables etc (although, I am very new to Make, so I could just be missing something).
In the below mock example, I’m just using JLD2 to directly save (and load) the output, but it’s the same concept as using
@produce_or_load in more complicated situations later - the saving and loading happens within the Julia scripts.
.PHONY: all all: tmp/single-sim_setup tmp/single-sim tmp/single-sim_setup: src/single-sim_setup.jl julia $^ @touch $@ tmp/single-sim: src/single-sim.jl tmp/single-sim_setup julia $^ @touch $@ .PHONY: clean clean: rm -rf tmp/* rm -rf data/singlesim/*
using DrWatson @quickactivate "TestCode" using JLD2 states = [100, 200, 300] time = 0:1:100 jldsave("data/singlesim/single-sim_setup.jld2"; states, time)
#%% using DrWatson @quickactivate "TestCode" using UnPack using JLD2 using Distributions using DataFrames #%% @unpack states, time = load("data/singlesim/single-sim_setup.jld2") #%% out = map(mean -> rand(Normal(mean), length(time), states) out_df = DataFrame(out, :auto) #%% jldsave("data/singlesim/single-sim_arrays.jld2"; out, out_df)
I originally used lots of chained
include() statements between scripts, but that made it difficult to develop multiple scripts that depended on the same source data, but had different requirements and tasks. I also wanted to make sure that everything was re-run when the dependencies were updated, and it was getting hard to keep track of which scripts to re-run.
If there is a better way of doing this (either with or without Make), please let me know. And if you need more information, I would be happy to expand upon anything.